Sunrise and Sunset Times by City

The follow plots show the amount of daylight, sunrise time, and sunset time for four US cities throughout the year of 2019.

Figure 1: Hours of Daylight, Sunrise Time, and Sunset Time by City

The rise and set times are shown in local time without adjustment for daylight saving times (DST). Data obtained from the U.S. Naval Observatory Astronomical Applications Department.

RangeMap: A Simple Interval Query Datastructure

Maps are a fundamental data structure. Their prevalence is a testament to their importance. Indeed, many search problems can be reduced to the construction of an appropriate map. However, a search problem occasionally arises that is difficult to solve, at least directly, with a map. The interval query problem is one such problem.

Continue reading “RangeMap: A Simple Interval Query Datastructure”

Visualizing International Trade of Food Products

With the rise of globalization, countries increasingly trade food products internationally. Acting in their own economic interests, countries buy and sell food where profitable, much like any other product. Import and export records provide a fascinating window into this complex world of international food trade.

Continue reading “Visualizing International Trade of Food Products”

Applying Correlation as a Criterion in Hierarchical Decision Trees

Decision trees are a simple yet powerful method of machine learning. A binary tree is constructed in which the leaf nodes represent predictions. The internal nodes are decision points. Thus, paths from the root to the leafs represent sequences of decisions that result in an ultimate prediction.

Decision trees can also be used in hierarchical models. For instance, the leafs can instead represent subordinate models. Thus, a path from the root to a leaf node is a sequence of decisions that result in a prediction made by a subordinate model. The subordinate model is only responsible for predicting samples that fall within the leaf.

This post presents an approach for a hierarchical decision tree model with subordinate linear regression models.

Continue reading “Applying Correlation as a Criterion in Hierarchical Decision Trees”

A Method for Addressing Nonhomogeneous Data using an Implicit Hierarchical Linear Model

Datasets containing nonhomogenous groups of samples present a challenge to linear models. In particular, such datasets violate the assumption that there is a linear relationship between the independent and dependent variables. If the data is grouped into distinct clusters, linear models may predict responses that fall in between the clusters. These predictions can be quite far from the targets depending on how the data is structured. In this post, a method is presented for automatically handling nonhomogenous datasets using linear models.

Continue reading “A Method for Addressing Nonhomogeneous Data using an Implicit Hierarchical Linear Model”

Decorrelating Features using the Gram-Schmidt Process

A problem that frequently arises when applying linear models is that of multicollinearity. The term multicollinearity describes the phenomenon where one or more features in the data matrix can be accurately predicted using a linear model involving others of the features. The consequences of multicollinearity include numerical instability due to ill-conditioning, and difficulty in interpreting the regression coefficients. An approach to decorrelate features is presented using the Gram-Schmidt process.

Continue reading “Decorrelating Features using the Gram-Schmidt Process”

TVLib: A C++ Text Vectorization Library with Python Bindings

I am a big fan of the CountVectorizer class in scikit-learn [1]. With a robust and easy interface that produces (sparse!) matrices, what’s not to love? Well, it’s… pretty… slow…

The performance is okay for 10s of MB of text, but GBs take minutes or more. It terms out that CountVectorizer is implemented in pure Python. The functions are single threaded too. It seems like low-hanging fruit. Just whip up some parallel C++, right? Well, not quite, but I’m getting ahead of myself.

Continue reading “TVLib: A C++ Text Vectorization Library with Python Bindings”