data science pt time series analysis.pptx

time analysis

1. Time Series Analysis A time series is a sequence of measurements from a system that varies in time. Collected data from a web site called \Price of Weed" that crowdsources market information by asking participants to report the price, quantity, quality, and location of cannabis transactions.

2. Dataset transactions = pandas.read_csv (mj-clean.csv, parse_dates =[5]) parse_dates tells read_csv to interpret values in column 5 as dates and convert them to NumPy datetime64 objects. The DataFrame has a row for each reported transaction and the following columns: city: string city name. state: two-letter state abbreviation. price: price paid in dollars. amount: quantity purchased in grams. quality: high, medium, or low quality, as reported by the purchaser. date: date of report, presumed to be shortly after date of purchase. ppg: price per gram, in dollars. state.name: string state name. lat : approximate latitude of the transaction, based on city name. lon : approximate longitude of the transaction.

3. Groupby Functions def GroupByQualityAndDay (transactions): groups = transactions.groupby (quality) dailies = {} for name, group in groups: dailies[name] = GroupByDay (group) return dailies def GroupByDay (transactions, func = np.mean ): grouped = transactions[[date, ppg]]. groupby (date) daily = grouped.aggregate ( func ) daily[date] = daily.index start = daily.date [0] one_year = np.timedelta64(1, Y) daily[years] = ( daily.date - start) / one_year return daily

4. Plotting The result from GroupByQualityAndDay is a map from each quality to a DataFrame of daily prices. thinkplot.PrePlot (rows=3) for i , (name, daily) in enumerate( dailies.items ()): thinkplot.SubPlot (i+1) title = price per gram ($) if i ==0 else thinkplot.Config ( ylim =[0, 20], title=title) thinkplot.Scatter ( daily.index , daily.ppg , s=10, label=name) if i == 2: pyplot.xticks (rotation=30) else: thinkplot.Config ( xticks =[])

5. Time series of daily price per gram for high, medium, and low quality cannabis

6. Linear Regression The following function takes a DataFrame of daily prices and computes a least squares fit, returning the model and results objects from StatsModels : def RunLinearModel (daily): model = smf.ols (ppg ~ years, data=daily) results = model.fit () return model, results iterate through the qualities and fit a model to each for name, daily in dailies.items (): model, results = RunLinearModel (daily) print(name) regression.SummarizeResults (results)

7. Results The following code plots the observed prices and the tted values: def PlotFittedValues (model, results, label=): years = model.exog [:,1] values = model.endog thinkplot.Scatter (years, values, s=15, label=label) thinkplot.Plot (years, results.fittedvalues , label=model)

8. Time series Analysis Most time series analysis is based on the modeling assumption that the observed series is the sum of three components: Trend: A smooth function that captures persistent changes. Seasonality: Periodic variation, possibly including daily, weekly, monthly, or yearly cycles. Noise: Random variation around the long-term trend.

9. Moving Averages A moving average divides the series into overlapping regions, called windows, and computes the average of the values in each window. One of the simplest moving averages is the rolling mean, which computes the mean of the values in each window .

10. Mean values pandas provides rolling_mean , which takes a Series and a window size and returns a new Series. >>> series = np.arange (10) array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9]) >>> pandas.rolling_mean (series, 3) array([ nan, nan, 1, 2, 3, 4, 5, 6, 7, 8])

11. Reindexing with rolling mean dates = pandas.date_range ( daily.index.min (), daily.index.max ()) reindexed = daily.reindex (dates) The first line computes a date range that includes every day from the beginning to the end of the observed interval. The second line creates a new DataFrame with all of the data from daily, but including rows for all dates, filled with nan. roll_mean = pandas.rolling_mean ( reindexed.ppg , 30) thinkplot.Plot ( roll_mean.index , roll_mean ) The window size is 30, so each value in roll_mean is the mean of 30 values from reindexed.ppg .

12. EWMA Approach An alternative is the exponentially-weighted moving average (EWMA),which has two advantages. First, as the name suggests, it computes a weighted average where the most recent value has the highest weight and the weights for previous values drop of exponentially. Second, the pandas implementation of EWMA handles missing values better. ewma = pandas.ewma ( reindexed.ppg , span=30) thinkplot.Plot ( ewma.index , ewma )

13. Daily price and a rolling mean (left) and exponentially-weighted moving average (right)

14. Missing values A simple and common way to fill missing data is to use a moving average. The Series method fillna does just what we want: reindexed.ppg.fillna ( ewma , inplace =True)