Minn Myat Soe :: His Notes

YQL is the way to get information from Web Services using SQL-like queries. It also provides us a console where we can test our queries and generate the REST query. To see how it works, just go to the console page, and enter the following as the YQL statement:

select * from yahoo.finance.quotes where symbol='FFIV'

And set the output to either “XML” or “JSON”, and click “Test”. I personally prefer JSON and will continue to use JSON throughout the example. I unchecked the “Diagnostics” and emptied the text field next to “JSON”.

The command will fetch the information related to the stock quote FFIV (F5 Networks, NASDAQ) from Yahoo Finance. Inside “Formatted View” window, you will see the result like this:

{
 'query': {
  'count': '1',
  'created': '2010-07-22T04:59:35Z',
  'lang': 'en-US',
  'results': {
   'quote': {
    'symbol': 'FFIV',
    'Ask': '80.00',
    'AverageDailyVolume': '1699250',
    'Bid': '77.63',
    'AskRealtime': '80.00',
    'BidRealtime': '77.63',
    'BookValue': '11.167',
    'Change_PercentChange': '-3.68 - -4.79%',
    'Change': '-3.68',
    'Commission': null,
    'ChangeRealtime': '-3.68',
    'AfterHoursChangeRealtime': 'N/A - N/A',
    'DividendShare': '0.00',
    'LastTradeDate': '7/21/2010',
    'TradeDate': null,
    'EarningsShare': '1.412',
    'ErrorIndicationreturnedforsymbolchangedinvalid': 'N/A',
    'EPSEstimateCurrentYear': '2.29',
    'EPSEstimateNextYear': '2.71',
    'EPSEstimateNextQuarter': '0.62',
    'DaysLow': '72.48',
    'DaysHigh': '77.74',
    'YearLow': '33.43',
    'YearHigh': '79.21',
    'HoldingsGainPercent': '- - -',
    'AnnualizedGain': '-',
    'HoldingsGain': null,
    'HoldingsGainPercentRealtime': 'N/A - N/A',
    'HoldingsGainRealtime': null,
    'MoreInfo': 'cnsprmiIed',
    'OrderBookRealtime': 'N/A',
    'MarketCapitalization': '5.859B',
    'MarketCapRealtime': null,
    'EBITDA': '188.9M',
    'ChangeFromYearLow': '+39.68',
    'PercentChangeFromYearLow': '+118.70%',
    'LastTradeRealtimeWithTime': 'N/A - <b>73.11</b>',
    'ChangePercentRealtime': 'N/A - -4.79%',
    'ChangeFromYearHigh': '-6.10',
    'PercebtChangeFromYearHigh': '-7.70%',
    'LastTradeWithTime': 'Jul 21 - <b>73.11</b>',
    'LastTradePriceOnly': '73.11',
    'HighLimit': null,
    'LowLimit': null,
    'DaysRange': '72.48 - 77.74',
    'DaysRangeRealtime': 'N/A - N/A',
    'FiftydayMovingAverage': '72.3831',
    'TwoHundreddayMovingAverage': '63.8515',
    'ChangeFromTwoHundreddayMovingAverage': '+9.2585',
    'PercentChangeFromTwoHundreddayMovingAverage': '+14.50%',
    'ChangeFromFiftydayMovingAverage': '+0.7269',
    'PercentChangeFromFiftydayMovingAverage': '+1.00%',
    'Name': 'F5 Networks, Inc.',
    'Notes': '-',
    'Open': null,
    'PreviousClose': '76.79',
    'PricePaid': null,
    'ChangeinPercent': '-4.79%',
    'PriceSales': '8.42',
    'PriceBook': '6.88',
    'ExDividendDate': 'N/A',
    'PERatio': '54.38',
    'DividendPayDate': 'N/A',
    'PERatioRealtime': null,
    'PEGRatio': '1.65',
    'PriceEPSEstimateCurrentYear': '33.53',
    'PriceEPSEstimateNextYear': '28.34',
    'Symbol': 'FFIV',
    'SharesOwned': null,
    'ShortRatio': '3.60',
    'LastTradeTime': '4:00pm',
    'TickerTrend': ' -===== ',
    'OneyrTargetPrice': '75.11',
    'Volume': '3213004',
    'HoldingsValue': null,
    'HoldingsValueRealtime': null,
    'YearRange': '33.43 - 79.21',
    'DaysValueChange': '- - -4.79%',
    'DaysValueChangeRealtime': 'N/A - N/A',
    'StockExchange': 'NasdaqNM',
    'DividendYield': null,
    'PercentChange': '-4.79%'
   }
  }
 }
}

Below that text field, we can find the REST statement which we can use to send query to the server. It looks like this for our query:

http://query.yahooapis.com/v1/public/yql?q=select%20*%20from%20yahoo.finance.quotes%20where%20symbol%3D'FFIV'%0A%09%09&format=json&env=http%3A%2F%2Fdatatables.org%2Falltables.env&callback=

This is how we can fetch stock information using YQL and receive information in JSON. Since this is a public data, we can directly send the REST, otherwise we need the API keys to access the data.

Let’s see how we can fetch the information via Python.

>>> import urllib2
>>> result = urllib2.urlopen('http://query.yahooapis.com/v1/public/yql?q=select%20*%20from%20yahoo.finance.quotes%20where%20symbol%3D'FFIV'%0A%09%09&format=json&env=http%3A%2F%2Fdatatables.org%2Falltables.env&callback=').read()
>>>  print result.read()

That should print the whole JSON response. We can use simplejson module to parse the result. It looks like this:

>>> import simplejson
>>> data = simplejson.loads(result.read())
>>> data['query']['results']['quote']['LastTradePriceOnly']
'2.35'

The python statements are pretty much self-explanatory.

Here’s another example from Yahoo, to get stock information from open data tables.

http://www.yqlblog.net/blog/2009/06/02/getting-stock-information-with-yql-and-open-data-tables/

In general, when I want to force the browser to access certain part of my website via https if the request is made with http, I would put a .htaccess inside that web directory.

RewriteEngine On
RewriteCond %{HTTPS} off
RewriteRule (.*) https://%{HTTP_HOST}%{REQUEST_URI}

But when I want to protect the directory with HTTP Basic Auth, it creates double authentication. I’ll expand this section after I captures the headers.

As a quick workaround, I use this hack in .htaccess

SSLOptions +StrictRequire
SSLRequireSSL
AuthUserFile /home/minn/.htpasswd
AuthType Basic
AuthName "Private Section"
Require valid-user
ErrorDocument 403 https://www.minnmyatsoe.com/private/

Haiku is another open source operating system, and IMO we can say it continues from where BeOS left off. I haven’t had a chance to try BeOS, but read about what it was supposed to do and some beautiful screenshots. BeOS was a closed source OS, and some loyal users tried to re-create the OS under OpenSource license.

And there came Haiku OS, an open source OS, and it released its alpha version on 09/2009. It is written in C++. The ISO image, as well as qemu/vmware images are now available to download. I just did a test run via their live CD image, and I would say I’m quite impressed. I wish it’d continue to R2 release soon.

I’ve moved this site over to the cloud servers, by Rackspace from my previous shared host. Actually I was looking for a cloud server and cloud space so that I can play with Hadoop. I found Amazon EC servers and S3, but their services charges are expensive for me. While searching for alternatives, CloudServers caught my attention.

It is cheaper than Amazon services, but at the moment I don’t think I can test Hadoop on CloudServer and with CloudSpace. I’m using it more like a virtual private server, that gives me “root” access. The good thing is you can modify the resources as you wish, so I would say it’s quite scalable. You are also charged by hours (uptime). Rackspace will also charge you even if you turn off the machine. They will not charge after we have deleted the server. If you want to test something for a project, you can just subscribe for desired amount of memory and disk space. And delete the server after it’s been used. We will only be charged for those period. That’s the flexibility that I prefer.

I’ll see what I can do with my server, and update the blog again.

Recently I’ve applied for entry visas for four of my relatives, about two or three weeks in advanced, and the visas have been approved by ICA.

Just one a day before the flight, one of my relatives has learnt that her daughter (2 years old) need to obtain visa and air tickets. I opened up the SAVE application website and made the application for the baby. Travel arrangements have already been made, and they were worried that the visa approval might be late, and they would have to rearrange the flights. It usually takes one business day to process. But to my surprise, ICA approved the visa within 3 hours after submission, and it has relieved all the worries of the families.

I really would like to give my heartfelt thank the officers at ICA who are working hard and understanding the need for urgency.

To those who want to submit visa application from the web:

• Online visa application can be found on the home page of ICA, which is http://www.ica.gov.sg
• Or do a seach on the google for “save singapore”, and follow the links.
• Please have your Singpass ready. If you don’t have one, it’s a good idea to make a request at http://www.singpass.gov.sg.
• The application will require a fee of S$30, which is payable by eNETS (Visa/Master)
• Please have the applicants information ready. Most data can be found on the passport, plus current address in home country, educational qualification. And a digital photo.