You can edit this page using wiki syntax for markup.

Article contents:[[tags:egg]] [[toc:]] == dataframe Tabular data structure for data analysis in CHICKEN Scheme, inspired by the data frame implementations in R, Python, and Racket, with an API modeled on dplyr's verbs (filter, group-by/summarize, joins, arrange, distinct, slice, sample, head/tail). == Documentation The {{dataframe}} library provides an interface for representing tabular data in rows and columns. It is inspired by the various dataframe implementations found in R, Python and Racket, and its row-filtering, grouping/summarizing, join, and row-selection operations are modeled on dplyr's verbs. The {{dataframe}} library also provides functions for loading and saving data from data frames as well as routines for descriptive statistics and linear regression. A data frame is a persistent (immutable) structure: every operation that "changes" one returns a new data frame and leaves the original untouched. === Installation <enscript highlight="bash"> chicken-install dataframe </enscript> This builds and installs four components: {{fmt-table}}, {{dataframe}}, {{dataframe-statistics}}, and {{dataframe-regression}}. === Columns Each dataframe consists of a collection of columns, which in turn is an object consisting of a unique key, data collection, and an associative list of properties. The following operations are defined on columns. <procedure>(column? obj)</procedure> Returns true if the given object is a column. <procedure>(column-key column)</procedure> Returns the key of the column. <procedure>(column-properties column)</procedure> Returns an associative list with column properties. <procedure>(column-collection column)</procedure> Returns the data collection of the column. <procedure>(column-deserialize column port)</procedure> Loads the data collection of a column from the given port. <procedure>(column-serialize column port)</procedure> Stores the data collection of a column to the given port in an s-expression format. === Creating data frames <procedure>(make-data-frame [column-key-compare: compare-symbol])</procedure> Creates a new dataframe, with optional argument a procedure that specifies how to compare column keys. Default is comparison on symbols ({{compare-int}} is also provided, for integer keys). Returns the new dataframe. <procedure>(df-insert-column df key collection properties)</procedure> Inserts a new column with the given key, data collection, and properties (or replaces the column, if {{key}} already exists). Returns a new dataframe with the inserted column. <procedure>(df-insert-derived df parent-key key proc properties)</procedure> Inserts a derived column, that is a column whose data elements are obtained by mapping a procedure onto the elements of an existing (parent) column. The derived column tracks its parent lazily, i.e. it is recomputed from the parent's current values rather than copied. Returns a new dataframe with the inserted column. <procedure>(df-insert-columns df lseq)</procedure> Inserts the columns contained in the given lseq of column objects. <procedure>(df-update-column df key collection properties)</procedure> Returns a new data frame with the existing column {{key}} replaced. <procedure>(df-delete-column df key)</procedure> Returns a new data frame with column {{key}} removed. <procedure>(df-from-rows column-keys source [column-key-compare: compare-symbol])</procedure> Creates a data frame with the given column keys and populates it with data from {{source}}, a list of rows (or a generator of rows), each row being a list of values in {{column-keys}} order. === Accessing data frames <procedure>(show df port)</procedure> Displays a table of the rows and columns contained in the dataframe to {{port}} ({{#f}} means the current output port). {{show}} is exported by the {{yasos}} egg, not by {{dataframe}} itself -- {{(import yasos dataframe)}} to use it. How many rows/columns are shown is controlled by the parameters below. <procedure>(display.max-elements)</procedure> <procedure>(display.max-columns)</procedure> Parameters controlling how many rows (default 20) and columns (default 10) {{show}} prints before truncating; set to {{0}} to disable truncation. <procedure>(df-row-count df)</procedure> Returns the number of rows in the dataframe. <procedure>(df-column df key failure-object)</procedure> Returns the {{(key . column)}} pair indicated by the given key, or {{failure-object}} if {{key}} isn't present. <procedure>(df-column-properties df key failure-object)</procedure> Returns the properties of column {{key}}, or {{failure-object}} if absent. <procedure>(df-collection df key failure-object)</procedure> Returns the data collection of column {{key}}, or {{failure-object}} if absent. <procedure>(df-columns df)</procedure> Returns a lazy sequence containing the columns of the dataframe. <procedure>(df-filter-columns df proc)</procedure> Returns a filtered lseq of the columns of the dataframe according to the given filter predicate procedure. <procedure>(df-select-columns df keys)</procedure> Returns an lseq of the columns of the dataframe that have the keys enumerated in the given list of keys. <procedure>(df-keys df)</procedure> Returns the keys of all columns in the dataframe. <procedure>(df-items df)</procedure> Returns an lseq of the key-column pairs contained in the dataframe. <procedure>(map-columns proc df [keys: #f])</procedure> Applies the given procedure to the named columns of the dataframe (default: every column) and returns the result as a dataframe. {{proc}} receives the column object itself. <procedure>(map-collections proc df [keys: #f])</procedure> Like {{map-columns}}, but {{proc}} receives each column's data collection rather than the column object. <procedure>(apply-collections proc df key ...)</procedure> Applies the given procedure to the data collections of the named columns of the dataframe and returns the result directly (i.e. not wrapped in a dataframe). <procedure>(reduce-collections proc df seed [keys: #f])</procedure> Fold over the data collections of the named columns, starting from {{seed}}. === Iterators <procedure>(df-for-each-column df proc)</procedure> Applies proc to each {{(key . column)}} pair. <procedure>(df-for-each-collection df proc)</procedure> Applies proc to the data collection of each column. <procedure>(df-gen-rows df)</procedure> Returns a generator procedure that returns the dataframe rows in succession, each row a list of values in {{df-keys}} order. <procedure>(df-gen-columns df)</procedure> Returns a generator procedure that returns {{(key . column)}} pairs in succession. === Row filtering <procedure>(df-filter-rows df predicate)</procedure> Returns a new data frame with only the rows for which {{predicate}} is true. {{predicate}} is called once per row with a ''row accessor'': a one-argument function that, given a column key, returns that column's value in the current row. <enscript highlight="scheme"> (df-filter-rows df (lambda (get) (> (get 'age) 30))) </enscript> Automatically picks between two strategies depending on {{df}}'s row count (rebuilding row by row for small data, or marking matches with a bit vector for large data -- see {{*filter-strategy-threshold*}} below); both give identical results, so this is a performance detail. <procedure>(df-filter-rows-multi df predicate ...)</procedure> Like {{df-filter-rows}}, but keeps rows for which ''every'' given predicate is true, evaluated together in a single pass. <procedure>(*filter-strategy-threshold*)</procedure> A parameter (default {{10000}}): {{df-filter-rows}} uses the bitmap strategy at or above this many rows, and the simpler row-rebuilding strategy below it. If {{df}} is grouped (see Grouping and summarizing below), the result stays grouped by the same columns. A group left with no rows after filtering simply doesn't appear, the same way dplyr's {{filter()}} treats a grouped tibble. === Grouping and summarizing <procedure>(df-group-by df group-keys)</procedure> Groups {{df}} by one column (a single key) or several (a list of keys). Returns a grouped data frame: it responds to every ordinary data-frame operation above ({{df-row-count}}, {{df-keys}}, {{show}}, and so on) by delegating to the underlying, ungrouped data, the same way a grouped tibble in dplyr is still a data frame. <procedure>(grouped-dataframe? obj)</procedure> Returns true if {{obj}} was returned by {{df-group-by}} (or any operation that preserves grouping). <procedure>(df-summarize grouped-df summary-specs)</procedure> Reduces a grouped data frame to one row per group. Each spec in {{summary-specs}} is {{(result-name source-col-key . function)}}: {{function}} is called with that group's values from {{source-col-key}}, as a plain list; if {{source-col-key}} is {{#f}}, {{function}} is called with the group's list of row indices instead, so a column-agnostic function like {{length}} gives the group's row count. <enscript highlight="scheme"> (df-summarize (df-group-by df 'category) (list (cons* 'n #f length) (cons* 'total 'amount (lambda (vs) (apply + vs))))) </enscript> The result is an ordinary (ungrouped) data frame, matching dplyr's {{summarise()}}, which likewise drops grouping once there's only one grouping variable left. <procedure>(df-group-by-apply df group-keys proc)</procedure> Shorthand for {{(proc (df-group-by df group-keys))}}. <procedure>(df-ungroup grouped-df)</procedure> Returns the data frame {{grouped-df}} was built from, discarding the grouping. === Joins Four dplyr-style joins, each {{(join left-df right-df by [suffix: '(".x" . ".y")])}}: <procedure>(df-inner-join left-df right-df by [suffix: '(".x" . ".y")])</procedure> Keeps rows whose join key is present in both data frames. <procedure>(df-left-join left-df right-df by [suffix: '(".x" . ".y")])</procedure> Keeps every row of {{left-df}}; {{right-df}}'s columns are filled with the symbol {{na}} where there's no match. <procedure>(df-right-join left-df right-df by [suffix: '(".x" . ".y")])</procedure> Keeps every row of {{right-df}}; {{left-df}}'s columns are filled with {{na}} where there's no match. <procedure>(df-full-join left-df right-df by [suffix: '(".x" . ".y")])</procedure> Keeps every row of both data frames, filling the other side with {{na}} where there's no match. {{by}} is one of: ; a symbol : join on that column, same name on both sides ; a list of symbols : join on all of them, same names on both sides ; a list of {{(left-key . right-key)}} pairs : for differently-named join columns, e.g. {{'((dept-id . id))}} The output's join-key column is always named after the left side's key and takes the left row's value when there is one, falling back to the right row's value (under its own name) for a right-only row in a right/full join. Duplicate keys fan out into the cross product of matches, as in a real relational join. Non-key columns that collide between the two sides are renamed with {{suffix}} (a {{(left . right)}} pair of strings, default {{(".x" . ".y")}}). <enscript highlight="scheme"> (df-inner-join users orders 'user-id) (df-left-join employees departments '((dept-id . id))) (df-inner-join sales targets '(date store-id) suffix: (cons ".a" ".b")) </enscript> === Row selection and ordering <procedure>(df-head df [n 6])</procedure> Returns the first {{n}} rows (default 6, as in R/dplyr); {{n}} is clamped to {{df}}'s row count. <procedure>(df-tail df [n 6])</procedure> Returns the last {{n}} rows. <procedure>(df-slice df indices)</procedure> Returns the rows at the given 0-based positions, in the order given. An index may repeat, duplicating that row. Row-index convention is 0-based, not R's 1-based. <procedure>(df-arrange df order-spec ...)</procedure> Sorts {{df}} by one or more columns. Each spec is a column key (ascending) or {{(key . 'desc)}} (descending); ties are broken by the next spec, and rows tied on every spec keep their original relative order. <enscript highlight="scheme"> (df-arrange df 'year (cons 'revenue 'desc)) </enscript> <procedure>(df-distinct df [columns] [keep-all: #f])</procedure> Returns the distinct rows of {{df}} (by default, distinct whole rows; {{columns}} restricts what counts as distinct), keeping the first occurrence of each and preserving their original relative order. Unless {{keep-all}} is true, only {{columns}} is kept in the result. <procedure>(df-sample df n [with-replacement: #f])</procedure> Returns {{n}} rows chosen at random: independently with replacement (duplicates possible) when {{with-replacement}} is true, or without replacement otherwise (errors if {{n}} exceeds {{df}}'s row count, since that many distinct rows don't exist). All six of the above preserve grouping: applied to a grouped data frame, the result is regrouped by the same columns. === Descriptive statistics From the {{dataframe-statistics}} component ({{(import dataframe-statistics)}}). Each of the following (other than {{describe}} and the {{grouped-*}} procedures) returns a data frame with the same columns as its input, each holding the single computed value for that column. <procedure>(describe df port)</procedure> Displays a table with the min/max/mean/sdev of each column in the dataframe. <procedure>(cmin df)</procedure> Computes the minimum value of each column. <procedure>(cmax df)</procedure> Computes the maximum value of each column. <procedure>(mean df)</procedure> Computes the mean value of each column. <procedure>(median df)</procedure> Computes the median value of each column. <procedure>(mode df)</procedure> Computes the mode value of each column. <procedure>(range df)</procedure> Computes the difference between maximum and minimum value of each column. <procedure>(percentile df)</procedure> Computes the percentile values of each column. <procedure>(variance df)</procedure> Computes the (sample) variance of each column. <procedure>(standard-deviation df)</procedure> Computes the (sample) standard deviation of each column. <procedure>(coefficient-of-variation df)</procedure> Computes the coefficient of variation of each column. <procedure>(grouped-mean grouped-df col-key)</procedure> Computes the mean of {{col-key}} within each group, as a data frame with one row per group (built on {{df-summarize}}). <procedure>(grouped-summary grouped-df col-key)</procedure> Computes the mean, min, max, standard deviation, and row count ({{n}}) of {{col-key}} within each group. === Regression and correlation From the {{dataframe-regression}} component ({{(import dataframe-regression)}}). <procedure>(linear-regression df x y)</procedure> Least-squares linear regression of column {{y}} on column {{x}}. Returns five values: intercept, slope, correlation coefficient {{r}}, {{R^2}}, and the significance of the slope (use {{let-values}} to capture all five). <procedure>(correlation-coefficient df x y)</procedure> Correlation coefficient between columns x and y (Pearson). <procedure>(spearman-rank-correlation df x y)</procedure> Spearman rank correlation coefficient between columns x and y. === I/O <procedure>(df-serialize df port)</procedure> Stores the dataframe in an s-expression format to the given port. <procedure>(df-deserialize df port)</procedure> Loads the data collections of the dataframe columns from the given port (typically called on a fresh {{(make-data-frame)}}) and returns the resulting data frame. === dplyr equivalents ; {{filter()}} : {{df-filter-rows}}, {{df-filter-rows-multi}} ; {{group_by()}} : {{df-group-by}} ; {{summarise()}} : {{df-summarize}} ; {{ungroup()}} : {{df-ungroup}} ; {{inner_join()}} : {{df-inner-join}} ; {{left_join()}} : {{df-left-join}} ; {{right_join()}} : {{df-right-join}} ; {{full_join()}} : {{df-full-join}} ; {{head()}} : {{df-head}} ; {{tail()}} : {{df-tail}} ; {{slice()}} : {{df-slice}} (0-based) ; {{arrange()}} : {{df-arrange}} ; {{distinct()}} : {{df-distinct}} ; {{slice_sample()}} : {{df-sample}} {{filter()}}/{{arrange()}}/{{slice()}}/{{distinct()}}/{{head()}}/{{tail()}} all preserve a grouped data frame's grouping, as in dplyr; {{summarise()}} drops it, also as in dplyr. Two real dplyr behaviors aren't implemented: {{filter()}}'s predicate can reference per-group aggregates in real dplyr (e.g. {{filter(x > mean(x))}} computed within each group); here, a predicate always sees a single row. And dplyr's {{slice()}} and grouped {{distinct()}} operate per group, whereas here they always operate on the whole table (then re-group the result) -- the same simplification applied uniformly across all six row-selection functions rather than singled out for just those two. == Examples <enscript highlight="scheme"> (import scheme srfi-1 yasos dataframe dataframe-statistics) (define df (make-data-frame)) (define df1 (df-insert-column df 'base (list-tabulate 100 (lambda (x) (- x 10))) '())) ;; exponential series (define df2 (df-insert-derived df1 'base 'exp (lambda (x) (* 2.0 (exp (* 0.1 x)))) '() )) (show df2 #f) (describe df2 #f) (linear-regression df2 'base 'exp) </enscript> == Testing <enscript highlight="bash"> csi -s tests/run.scm </enscript> == About this egg === Author [[/users/ivan-raikov|Ivan Raikov]] === Repository [[https://github.com/iraikov/chicken-dataframe|https://github.com/iraikov/chicken-dataframe]] === Version history ; 1.0 : Added dplyr-style operators for joins, filters selection; ported to CHICKEN 6. ; 0.1 : Initial release === License Copyright 2019-2026 Ivan Raikov. This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. A full copy of the GPL license can be found at <http://www.gnu.org/licenses/>. Description of your changes:

I would like to authenticate

Spam control

What do you get when you subtract 11 from 17?