How does hive UDF obtain IDC/ provinces / cities through IP addresses
This article will explain in detail how hive UDF obtains IDC/ provinces / cities through IP addresses. The editor thinks it is very practical, so I share it with you as a reference. I hope you can get something after reading this article.
Brief description:
A simple binary search algorithm locates the IP segment to which the IP belongs according to the IP address, and then obtains the information of the IDC/ province / city of the IP segment.
Input: IP geographic information file. After getting the address library, you need to format it. Refer to:
1. If the province is null or'', city is null or''= "province = other and city = other
two。 The province is not empty and is a municipality directly under the Central Government, but the city is not a municipality directly under the Central Government = "city = municipality directly under the Central Government"
3. The province is not empty and is not a municipality directly under the Central Government, but the city is empty = "city = other"
4. Illegal information in provinces or cities such as (,\) = "provinces = other and cities = other
/ user/hadoop/IP.csv
Format:
Serial number, start IP (long), end IP (long), province, city, IDC, start IP, end IP 29990meme 16777472 people 16778239, Fujian Province, other, Telecom, 1.0.0.255 29991Min 16779264pint 16781311, Guangdong Province, others, Telecom, 1.0.8.0pr 1.0.255 29992pr 16785408mein 16793599, Guangdong Province, others, Telecom, 1.0.32.0pm 1.0.63.255
Usage & output:
Refer to this article for editing, packaging or compiling to hive. I won't say much about it here: http://my.oschina.net/wangjiankui/blog/64230
Get_ip_location_new (visitip,'IDC') / / returns IDC information
Get_ip_location_new (visitip,'REGION') / / returns province information
Get_ip_location_new (visitip,'CITY') / / returns city information
Code:
Package com.xxx.hive.udf;import java.io.BufferedReader;import java.io.IOException;import java.io.InputStreamReader;import java.net.URI;import java.util.ArrayList;import java.util.HashMap;import java.util.List;import java.util.Map;import java.util.StringTokenizer;import org.apache.hadoop.conf.Configuration;import org.apache.hadoop.fs.FSDataInputStream;import org.apache.hadoop.fs.FileSystem;import org.apache.hadoop.fs.Path;import org.apache.hadoop.hive.ql.exec.UDF Import org.apache.hadoop.io.IOUtils;import org.apache.hadoop.io.Text;public class UDFGetIPLocationNew extends UDF {public static List map = new ArrayList (); public static long [] start_from_index; public static long [] end_to_index; public static Map idcCache = new HashMap (); public static Map regionCache = new HashMap (); public static Map cityCache = new HashMap (); private void LoadIPLocation () {Configuration conf = new Configuration (); String namenode = conf.get ("fs.default.name") String uri = namenode + "/ user/hadoop/IP.csv"; FileSystem fs = null; FSDataInputStream in = null; BufferedReader d = null; try {fs = FileSystem.get (URI.create (uri), conf); in = fs.open (new Path (uri)); d = new BufferedReader (new InputStreamReader (in)); String s = null; while (true) {s = d.readLine () If (s = = null) {break;} map.add (s);}} catch (IOException e) {e.printStackTrace ();} finally {IOUtils.closeStream (in);}} public static int binarySearch (long [] start, long [] end, long ip) {int low = 0; int high = start.length-1 While (low = start [middle]) & (ip
