Big data analysis in people search, Really challenging.

Nowadays, more and more people, from both industry and academy, are talking about big data. its popularity is like the fanaticism brought by MJ's new album if he's still alive. It is no exaggerating to say that we are now living in an era of "Big Data": science, engineering and technology are producing increasingly large data streams, with petabyte and exabyte scales becoming increasingly common. Big data presents opportunities and also perils. On the optimistic side, it gives us a pretty good opportunity to scale existing theoretical principles and also learning algorithms from modest-size data sets to massive data sets, thus, big data give us a fantastic platform and resources to verify the success of those principles and algorithms; while on the other side, there are some big challenges also brought by big data. It amplify the errors and their effects in existing technologies, and also, efficiency, space, energy are major issues we need to tackle in big data analysis. 

As the social network website are booming since later 90s and early 00s, millions of people join these kind of website like facebook and twitter each year, as the official statistic data of facebook says, it has a billion active user at the end of 2012. People are looking for and sharing all kinds of information on these website, and an obvious trend leaded by social network is that people are much more willing to post and share their personal information online. As a result, anyone wants to find his friends and try to get their recent updates, he can get all of these information by just typing his friends' names online, for example, on google or on some social network websites. Moreover, as this kind of requirement becomes more popular and general, an advanced people search engine is very useful. 

Recently, I'm investigating people search. I felt that neither google or social network website like facebook cannot exactly meet my requirement, actually, there is a wonderful people search engine which can aggregate a person's profile around the internet, not only from social network website, but also from all kinds of personal pages, even news pages online. Its name is called "whova",  provides advanced people search ability which is built upon proprietary big data analytics and mining technology. To precisely aggregate one's personal information online is very challenging, because:

1) Firstly, as we have mentioned above, a single person's information can be scattered around many websites, for example, I may ever submitted my personal information, such as, name, education, affiliation etc on facebook, and also on other website, eg linkedin. To precisely determine that different profiles information parsed from different website belong to a same person is very difficult.

2) Secondly, it may be some kind easy to get profile information from social network website, since the information show on these websites are structural. However, for general pages, an intelligent parser must be carefully designed to inference personal profile from those complex text paragraphs on a page, as we know, semantic analysis is a pretty challenging open problem.

Hbase Setup in Ubuntu

Apache HBase is the Hadoop database, which is a distribute, sacalable, big data store. The following is some steps of configuring and using HBase as a new user guide.

1) Firstly, download latest version of HBase from official site: http://hbase.apache.org/. A binary version is preferred.

2) Unpack the compressed tar file, and start to config it. The most important things of configuring hbase includes:
i) Change conf/hbase-site.xml to look like this:

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
  <property>
    <name>hbase.rootdir</name>
    <value>file:///DIRECTORY/hbase</value>
  </property>
  <property>
    <name>hbase.zookeeper.property.dataDir</name>
    <value>/DIRECTORY/zookeeper</value>
  </property>
</configuration>

while, DIRECTORY is your local directory, which you want to store the data, a sample name could be like this: /home/yourname/database/hbase. And also, you can set up the directory for zookeeper, but that's not mandatory.

ii) For ubuntu users, or even linux users, also you should change your hosts a little bit. As the tutorial in the official site says: "HBase expects the loopback IP address to be 127.0.0.1. Ubuntu and some other distributions, for example, will default to 127.0.1.1 and this will cause problems for you". A modified hosts file possibly may look like this:

127.0.0.1 localhost

127.0.0.1 yourname

127.0.1.1 yourname

# The following lines are desirable for IPv6 capable hosts

::1     ip6-localhost ip6-loopback

fe00::0 ip6-localnet

"yourname" means your user name in ubuntu.  If you do not modify the hosts file in /etc, then possibly, you may encounter a problem when you start to use hbase shell: HBase will give you a error message says "ERROR: org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing". The loopback IP address was set up correctly, which may cause the master to stuck in the initialization process.

3) Then you can simply launch HBase by using: ./bin/start-hbase.sh. The master will be running, you can check it by looking at running process in your ubuntu. You can also terminate HBase by using: ./bin/stop-hbase.sh

4) Moreover, you can also use HBase shell to create new tables or operate on existed tables by using: ./bin/hbase shell. But firstly, you should launch HBase master by using the command in step 3), otherwise, HBase shell is not gonna work.

Just Do It

Sometime, something, you just need to make the first step, then everything should be much easier for you.

About Me

About me:


I am now a post-doc in the University of California, San Diego, since Aug. 2013. Before joining Opera group, I graduated from Institution of Computing Technology, Chinese Academy of Sciences with PhD in computer science. I ever studied in the University of Science and Technology of China with a major in EE from 2003 to 2007.

My interests lie in many fields, including: advanced compiler, program analysis, data mining, machine learning, web development. Including C, C++, Java, python, perl are used during my daily development. I am also interested in big data analysis, for instance, effectively and efficiently collecting and analyzing people connection information from the internet.