Hong Kong property data can work for you: here’s how

Buying a property will — for most people — be the biggest purchase they will ever make. So what’s the smart way for a data scientist to figure out the best flat deal in Hong Kong? Scrape and hack the market of course.

Normal home buyers may burn a lot of shoe leather visiting dozens of real estate agents, spend hours on websites looking at individual entries, maybe even start a spreadsheet.

But using a bit of data, or in this case around 1.65 million transaction records, we start to see through the sales pitches and get a feel for the total market.

Alternatively, there are many websites with dozens of pages as research fodder.

hk1to90
Only 90 pages, no biggie

What’s missing is a tool that allows you to view the actual sale prices throughout Hong Kong with a single user interface.  That’s what we developed during Data Science Hong Kong’s fifth unhackathon.

A market exploration tool

Using a dataset of 1.65 million transactions scraped from the site of one of Hong Kong’s main real estate agencies, this tool (pictured below) shows the average price per square foot by location and time.

new_screen
A visual representation of property price per square foot in 2017

Reading the data

Each circle represents a building, colour-coded and shaded darker to show higher prices per square foot. Where more sales are registered in the user’s chosen time period the circle grows larger.

With this tool, market understanding becomes a lot easier and more intuitive. In one single interface, all the actual transactions can be summarized and filtered using attributes like price, size, address or sale date — for example, flats between 300 and 400 square feet.

At first glance, the colours of the markers would suggest that Central and the west of Hong Kong Island look like the most expensive areas, followed by Tsim Sha Tsui, and the southern Kowloon peninsula.

Looking closer at the northern edge of Kowloon Park, we can set the transaction year on 2017. A summary appears upon hovering over the property and we can see that 3 flat sales occurred at 10-24 Parkes Street (known as Wing Fu Mansion), for an average price of under HK$14,400 a square foot.

focus_example
Sales in the north of Kowloon Park in 2017 

The following animation shows how easy it is to navigate through the districts and market history, and get a clear and visual idea of prices in the area.

js_research_red
Only the beginning for our property search tool

You can try it yourself by following this link.

How to get the data

To build this tool, we need first to collect the data.

transaction_sample
How the original data was presented online

This dataset was scraped using the Python webscraping library Scrapy. For most transactions, it has the flat, floor area, floor and building address and most importantly the prices and dates of sales.

json_line
One transaction in JSON format

After cleaning the data, our team began looking at prices and transaction volumes, including outliers. You can see a presentation of the data here.

Collecting geolocation data

To be able to correctly place a flat on a map, we need its geolocation, consisting of a longitude and latitude pair. The only location data we have is the address, so we can use the Google Maps API to convert the addresses to lat/lon pairs.

But Google limits API calls at 2500 addresses, with a 50 cent charge for an additional 1000 requests. Small change, but fortunately it’s not necessary in Hong Kong as there are other organizations that also offer this service.

The OFCA government website, used to check whether a building has digital television or fibre-optic broadband, also returns the geolocation of the searched address. We automatically requested all block addresses. In the following screenshots, we verify correct resolution of the address.

resu_robinson
Geolocation results for 103 Robinson Road
robinson_road
Google Map results for this geolocation
number
Street view on the result: building # 103

After joining transactions and geolocations, we can start building the tool.

Building the visualization tool

The main steps to build  this tool are:

  • Creating a basic html template with text blocks and containers for titles, caption, transaction year filter and popup text
  • Loading the map from the ESRI javascript API. The parameters are set to focus on Hong Kong.
  • Loading the transaction data and looping over it . For each year and address, we compute the circle parameters.
    • The radius is a linear function of the square root of transaction quantities. With this choice, circle area and transaction quantity are linearly dependent.
    • The colors (red and green channels) are simple linear functions of the price.
  • Managing user events: When the user hovers the mouse on a point, a javascript function is called to display the data related to the point. When the mouse leave, the data is cleared.

Taking the next steps

Various improvements to this first version can be implemented:

  • Some data quality issues have been raised (missing flat areas, block address, etc.) and need to be corrected
  • The decision of buying a flat includes other factors that can be incorporated into this visualization tool. For instance, car traffic, public transport options, school networks (some of which are already included in the data), average pollution levels, altitude/topography
  • Adding current flat sale offers in the vicinity to help find the best deals

If you have any question or want to further explore the data, don’t hesitate to send a message in the slack group: datasciencehk.slack.com

Credits:
Thanks go to the Data Science Hong Kong organizers for this event.

Data science news round up

Our tight-knit community of data scientist have shared a wealth of news and inspiring projects from around the web over the past couple of months. Here is a brief round up of the more interesting articles, and remember, you can join in on our slack group.

2-l-304106-unsplash

Millions of Chinese farmers reap benefits of huge crop experiment

An article that demonstrates the world changing potential of evidence based approaches to the world’s problems. For me, it’s also a reminder that it’s often not the latest buzzword or most glamourous topics that have the most impact.

Winning with Data Science

Next is an article examining the business and organisational side of data science. This is a topic that probably doesn’t get enough attention compared to the latest and coolest algorithm. It’s important for data scientists to take an interest in how organisations should adapt, if they don’t it will probably be decided by someone not qualified to make the decision!

nasa-43569-unsplash

What Comes After Deep Learning?

This article examines whether deep learning is actually a blind alley and considers what new approaches might be next for data science. Also a brief examination of the question of US vs China in the AI “arms race”.

‘Who’s Leading AI’ Isn’t the Intelligent Question

Our final article explores the much talked about question of whether the US or China is winning and why it’s not the right question to ask.

If you found any of these articles interesting then do come and join the discussion on our Slack group, where you will also find details of meetups. https://datasciencehk.slack.com/

Women in data science – WiDS 2018

The Stanford Women in Data Science conference 2018  is starting on March 6th at 1am Hong-Kong time

Live Broadcast

We encourage everyone to follow the broadcast here 

You can tweet using the hashtag #WiDS2018Q

Program

The program can be found here, we reproduce it here for convenience in HK time zone

1:00-1:10am: Opening Remarks: Margot Gerritsen, Senior Associate Dean and Director of ICME, Stanford University
1:10-1:30am: Welcome Address: Maria Klawe, President, Harvey Mudd College
1:30-2:05am: Keynote Address: Leda Braga, CEO, Systematica Investments
2:05-2:10am Regional Event Check-in
2:10-2:50am: Technical Vision Talks:
     2:10-2:30am Mala Anand, EVP, President, SAP Leonardo Data Analytics
     2:30-2:50am Lada Adamic, Research Scientist Manager, Facebook
2:50-3:10am: Morning break
3:10-3:15am: WiDS Datathon Winners Announced
3:15-3:55am: Technical Vision Talks:
     3:15-3:35am: Nathalie Henry Riche, Researcher, Microsoft Research
     3:35-3:55am: Daniela Witten, Associate Professor of Statistics and Biostatistics, University of Washington
3:55am-4:30am: Keynote Address: Latanya Sweeney, Professor of Government and Technology in Residence, Harvard University
4:30-6:00am:  Lunch and Breakouts (NO LIVESTREAM)
6:00-6:35am: Keynote Address: Jia Li, Head of Cloud R&D, Cloud AI, Google
6:35-7:15am Technical Vision Talks:
     6:35-6:55am: Bhavani Thuraisingham,
Professor of Computer Science and Executive
Director of Cyber Research and Education Institute, University of Texas at Dallas
     6:55-7:15am: Elena Grewal, Head of Data Science, Airbnb
7:15-7:30am  Afternoon break 

7:30-7:35am Regional event check-in
7:35-8:15am Career Panel moderated by Margot Gerritsen
Bhavani Thuraisingham 
 Professor of Computer Science and Executive
Director of Cyber Research and Education Institute, University of Texas at Dallas
     Ziya Ma,  Vice President of Software and Services Group and Director of Big Data Technologies, Intel Corporation
     Elena Grewal Head of Data Science, Airbnb
     Jennifer Prendki, Head of Data Science, Atlassian
8:15-8:55am: Technical Vision Talks
     8:15-8:35am: Risa Wechsler, Associate Professor of Physics, Stanford University
     8:35-8:55am: Dawn Woodard, Senior Data Science Manager of Maps, Uber
8:55-9:00am: Closing Remarks