Compare commits

...

59 Commits

Author SHA1 Message Date
9692d7c100 Nombres funciona completo sobre faaso 2024-08-21 10:36:32 -03:00
9821b58cce Busqueda con estilo 2024-08-20 12:16:36 -03:00
8509a15f68 Busqueda con estilo 2024-08-20 12:14:08 -03:00
e8483662e1 debug code push 2024-08-20 10:38:38 -03:00
33c95bca6b Switching the crystal functions to faaso 2024-08-20 10:33:21 -03:00
8846ad3841 lint 2024-05-16 16:30:05 -03:00
3b1069df72 Fix compound names in historico 2024-05-16 16:25:34 -03:00
f2ee8aa6e8 Prepared statements everywhere 2024-05-16 15:31:09 -03:00
76e16ca803 Make split by gender faster 2024-05-16 10:00:14 -03:00
e4c44dcae1 log query 2024-05-15 20:35:26 -03:00
1733aa40df log query 2024-05-15 20:32:17 -03:00
49a486af66 log query 2024-05-15 20:28:45 -03:00
652f5838b6 log query 2024-05-15 20:27:47 -03:00
3f1b41c316 longer timeout in busqueda 2024-05-15 20:18:09 -03:00
9283ee0a03 Fix datatypes in query 2024-05-15 20:06:35 -03:00
6ed644d1d0 Fixing stuff 2024-05-15 19:58:51 -03:00
6d7a571a55 Fixing c-busqueda 2024-05-15 19:47:54 -03:00
d186fc3d68 Starting port of c-busqueda, not finished 2024-05-15 18:07:25 -03:00
02e7594e3d Port c-historico to postgres 2024-05-15 18:02:50 -03:00
b0fafae4ef Tweaks 2024-04-08 14:58:47 -03:00
9aca3f2e14 add OPENFAAS_URL to build.sh 2024-03-22 16:34:21 -03:00
3173c610bf IOL: More aggressive caching, even if token fails 2023-08-31 15:17:58 -03:00
af94db7d4e Remove python versions of nombres functions 2023-08-30 11:05:04 -03:00
abfe0cbf27 Funcion repetida 2023-08-07 12:21:54 -03:00
338944d82c Devolver ultimo valor si IOL se cae 2023-08-07 12:14:16 -03:00
3fa1975c69 Add secret management 2023-07-17 20:08:45 -03:00
a1ffdaf758 iol 2023-07-02 09:55:16 -03:00
f059e48ce2 Mas instrumentos 2023-07-01 21:50:44 -03:00
f0ab7c90dd Initial iol 2023-07-01 19:59:14 -03:00
f067d97095 Removed unused template 2023-06-06 13:43:35 -03:00
4af99590ed Mejor yrange en c-busqueda 2023-06-05 16:02:54 -03:00
cddccc416f Updated README 2023-06-04 20:34:16 -03:00
52007a4b5b Handle sparse results 2023-06-04 19:34:45 -03:00
6a357fe776 Fix year handling 2023-06-04 19:08:36 -03:00
9263815727 * Misma funcion query que la otra funcion
* Mejor soporte de nombres que no estan en la base de datos
2023-06-04 18:17:22 -03:00
a55784a4cd Valor default para title 2023-06-04 18:16:21 -03:00
4c46cb2958 Sync format_buffer code 2023-06-04 17:43:51 -03:00
823cc24e8a Typo, usar miles de verdad 2023-06-04 17:39:03 -03:00
49352c8e37 * Ignore composite names
* Show guesses
2023-06-04 15:30:30 -03:00
bfe2029b2d Show error traces when building 2023-06-04 15:29:40 -03:00
e17bd4e6aa Move things around so the shards layer is cached 2023-06-04 13:16:51 -03:00
3bbafb28c0 Long timeout until I optimize things 2023-06-04 11:54:22 -03:00
b4d66769b3 Initial implementatin of busqueda in crystal 2023-06-04 11:53:24 -03:00
d2f590665d Unique canvas name suppor 2023-06-04 11:52:30 -03:00
aeb18bf928 Ignore shard.lock 2023-06-04 11:51:43 -03:00
a62d814aaf Added missing file, refactored normalize_name out 2023-06-03 21:18:50 -03:00
cafe5fdce5 Historico ported to crystal 2023-06-03 21:09:25 -03:00
f7a54c1d7a Alpine-based crystal-http template 2023-06-02 18:18:46 -03:00
d48cabc84f Nicer/smaller crystal template 2023-06-02 17:21:10 -03:00
0e5a41c170 Not going to reimplement tapas in crystal 2023-06-01 20:11:40 -03:00
ad164ac7f2 Make image smaller, use the correct platform 2023-06-01 19:32:12 -03:00
94ea99983c Support passing --filter or whatever 2023-06-01 19:31:38 -03:00
1cc3ef2e82 Changed crystal template, picked latest flask one 2023-06-01 18:52:39 -03:00
f2be91d628 Testing functions in crystal. Had to commit the template because it was outdated 2023-06-01 09:44:33 -03:00
0045b432b4 (c) date 2023-05-16 17:46:21 -03:00
ad57ba69b4 Remove unused test boilerplate 2023-05-16 15:48:37 -03:00
6340c3e3d0 Renamed yaml file 2023-05-16 15:29:12 -03:00
3cb29002df Handle empty requests for ping, add a bit of styling 2023-05-16 14:01:04 -03:00
b520913c47 remove comment 2023-05-16 12:58:53 -03:00
53 changed files with 1791 additions and 371 deletions

8
.gitignore vendored
View File

@@ -1,3 +1,4 @@
.faaso.yml
# Byte-compiled / optimized / DLL files
__pycache__/
*.py[cod]
@@ -161,4 +162,9 @@ cython_debug/
.venv
.vscode/
template/
build
.secrets
shard.lock
template

View File

@@ -1,6 +1,6 @@
MIT License
Copyright (c) 2020 Roberto Alsina
Copyright (c) 2020-2023 Roberto Alsina
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal

View File

@@ -9,6 +9,10 @@ En particular faas-cli (ver scripts build.sh y deploy.sh) necesita docker con bu
Funciones para hacer cosas con la data de nombres de Argentina.
Las que tienen prefijo `c-` están reescritas en Crystal, para practicar
el lenguaje pero de hecho funcionan mejor que las otras y son las que están en
este momento en producción.
Página con esto funcionando: http://nombres.ralsina.me
## tapas
@@ -16,3 +20,7 @@ Página con esto funcionando: http://nombres.ralsina.me
Funcion para generar tapas de libros
Página con esto funcionando: http://covers.ralsina.me
## iol
Proxy para acceder a la api de IOL de manera más sencilla.

View File

@@ -6,10 +6,15 @@ docker run --rm --privileged \
multiarch/qemu-user-static \
--reset -p yes
# Build and deploy
if [ ! -d template ]
then
faas-cli template store pull python3-flask
fi
faas-cli publish -f nombres.yml --platforms linux/arm64 --build-arg 'TEST_ENABLED=false'
faas-cli deploy -f nombres.yml
# secrets
export OPENFAAS_URL=http://pinky:8082
pass iol-pass | faas-cli secret create iol-pass
pass iol-user | faas-cli secret create iol-user
pass iol-api-secret | faas-cli secret create iol-api-secret
pass nombres-user | faas-cli secret create nombres-user
pass nombres-pass | faas-cli secret create nombres-pass
faas-cli publish -f functions.yml --platforms linux/arm64 --build-arg 'TEST_ENABLED=false' $*
faas-cli deploy -f functions.yml $*

11
busqueda/README.md Normal file
View File

@@ -0,0 +1,11 @@
# Readme for Busqueda
This is a funko using the Crystal runtime for [FaaSO](https://git.ralsina.me/ralsina/faaso)
## What is Busqueda
Write here what it is
## How to use Busqueda
And so on.

149
busqueda/funko.cr Normal file
View File

@@ -0,0 +1,149 @@
require "json"
require "kemal"
require "pg"
require "pool/connection"
# get credentials from secrets
USER = File.read("/secrets/user").strip
PASS = File.read("/secrets/pass").strip
DBHOST = File.read("/secrets/dbhost").strip
DBURL = "postgres://#{USER}:#{PASS}@#{DBHOST}:5432/nombres"
puts "Connnecting to #{DBURL}"
# Create a connection pool to the database
pg = ConnectionPool.new(capacity: 5, timeout: 1.seconds) do
PG.connect(DBURL)
end
def normalize(s : String) : String
s.unicode_normalize(:nfkd)
.chars.reject! { |character|
!character.ascii_letter? && (character != ' ')
}.join("").downcase
end
# A basic hello world get endpoint
post "/" do |env|
prefijo = env.params.json["p"].as(String)
genero = env.params.json["g"].as(String)
year = env.params.json["a"].as(String)
prefijo = normalize(prefijo)
if !["f", "m"].includes?(genero)
genero = nil
end
datos = [] of Tuple(String, Int32 | String)
# Connect using credentials provided
pg.connection do |cursor|
if prefijo.empty? && year.empty?
result_set = cursor.query("
SELECT nombre, total::integer
FROM totales
ORDER BY total DESC
LIMIT 50")
elsif prefijo.empty? && !year.empty?
# Per-year totals
result_set = cursor.query("
SELECT nombre, contador::integer
FROM nombres
WHERE
anio = $1
ORDER BY contador DESC
LIMIT 50", year)
elsif !prefijo.empty? && year.empty?
# Filter only by prefix
result_set = cursor.query("
SELECT nombre, total::integer
FROM totales
WHERE
nombre LIKE $1
ORDER BY total DESC
LIMIT 50", prefijo + "%")
elsif !prefijo.empty? && !year.empty?
# We have both
result_set = cursor.query("
SELECT nombre, contador::integer
FROM nombres
WHERE
anio = $1 AND
nombre LIKE $2
ORDER BY contador DESC
LIMIT 50", year, prefijo + "%")
end
if !result_set.nil?
result_set.each do
nombre = result_set.read(String)
valor = result_set.read(Int32)
datos.push({nombre, valor})
end
result_set.close
end
if datos.empty?
raise "No data found"
end
end
# In this context, remove all composite names
datos.reject! { |row|
row[0].to_s.includes? " "
}
datos.insert(0, {"Nombre", "Cuantos?"})
if genero
pg.connection do |cursor|
datos.reject! { |row|
# How feminine is this name?
# Yes this database is upper case
puts "Checking #{row[1]} #{row[0]}"
feminidad = 0
sql = %(
SELECT COALESCE((SELECT frecuencia FROM mujeres WHERE nombre='#{row[0]?.to_s.upcase}'), 0) AS mujeres,
COALESCE((SELECT frecuencia FROM hombres WHERE nombre='#{row[0]?.to_s.upcase}'), 0) AS hombres
)
puts "SQL: #{sql}"
cursor.query sql do |result_set|
result_set.each do
mujeres = result_set.read(Int32)
hombres = result_set.read(Int32)
puts "frecuencias: #{mujeres} #{hombres}"
if hombres == mujeres == 0
feminidad = 0.5
else
feminidad = mujeres / (hombres + mujeres)
end
end
end
# El overlap en 0.5 es intencional!
if (feminidad >= 0.5 && genero == "f") ||
(feminidad <= 0.5 && genero == "m")
false
else
true
end
}
puts "Data split by gender"
end
end
datos = datos[..10].map { |row|
[row[0].capitalize, row[1]]
}
if datos.size > 2
title = "¿Puede ser ... #{datos[1][0].to_s.titleize}? ¿O capaz que #{datos[2][0].to_s.titleize}? ¡Contame más!"
elsif datos.size == 2
title = "Me parece que ... #{datos[1][0].to_s.titleize}!"
else
title = "No tengo idea!"
end
{
"title" => title,
"data" => datos,
}.to_json
end
get "/ping/" do
pg.connection.exec("SELECT 42")
"OK"
end

11
busqueda/funko.yml Normal file
View File

@@ -0,0 +1,11 @@
name: busqueda
runtime: kemal
options:
shard_build_options: "--release -d --error-trace"
ship_packages: []
devel_packages: []
healthcheck_options: "--interval=1m --timeout=2s --start-period=2s --retries=3"
healthcheck_command: "curl --fail http://localhost:3000/ping || exit 1"
copy_from_build:
- "public public"
- "bin/funko ."

View File

@@ -1,164 +0,0 @@
import unicodedata
import urllib
from collections import namedtuple as nt
from dataclasses import dataclass
from json import loads
import logging
import pygal
import pyrqlite.dbapi2 as dbapi2
import requests
connection = dbapi2.connect(
host="10.61.0.1",
user="root",
port=4001,
password="",
)
def remove_accents(input_str):
nfkd_form = unicodedata.normalize("NFKD", input_str)
return "".join([c for c in nfkd_form if not unicodedata.combining(c)])
def femininidad(nombre):
sql1 = """
SELECT COALESCE(frecuencia,0)
FROM mujeres WHERE nombre=:nombre
"""
sql2 = """
SELECT COALESCE(frecuencia,0)
FROM hombres WHERE nombre=:nombre
"""
with connection.cursor() as cursor:
nombre = nombre.strip().split()[0].strip().upper()
mujeres = cursor.execute(sql1, {"nombre": nombre}).fetchone()
mujeres = 0 if mujeres is None else mujeres[0]
hombres = cursor.execute(sql2, {"nombre": nombre}).fetchone()
hombres = 0 if hombres is None else hombres[0]
if hombres == mujeres == 0:
return 0.5
return mujeres / (hombres + mujeres)
def split_por_genero(nombres):
femeninos = []
masculinos = []
for n in nombres:
fem = femininidad(n[1])
if fem is None:
femeninos.append(n)
masculinos.append(n)
elif fem >= 0.5:
femeninos.append(n)
else:
masculinos.append(n)
return {"f": femeninos, "m": masculinos}
def handle(req):
"""handle a request to the function
Args:
req (str): request body
{
p: prefijo del nombre,
g: genero del nombre,
a: año de nacimiento
}
"""
try:
data = loads(req)
except Exception as e:
data = {}
try:
prefijo = data.get("p") or None
genero = data.get("g") or None
try:
año = int(data.get("a"))
except Exception:
año = None
except Exception as e:
prefijo = genero = año = None
return f"{req} -- {e}", 400
if prefijo is not None:
prefijo = prefijo.strip().lower()
if genero not in ("f", "m"):
genero = None
if prefijo is None and año is None: # Totales globales
with connection.cursor() as cursor:
sql = """
SELECT total, nombre
FROM totales
ORDER BY total DESC
LIMIT 50
"""
cursor.execute(sql)
datos = [(r["total"], r["nombre"]) for r in cursor.fetchall()]
elif prefijo is None and año is not None: # Totales por año
with connection.cursor() as cursor:
sql = """
SELECT contador, nombre
FROM nombres
WHERE
anio = :anio
ORDER BY contador DESC
LIMIT 50
"""
cursor.execute(sql, {"anio": año})
datos = [(r["contador"], r["nombre"]) for r in cursor.fetchall()]
elif prefijo is not None and año is None:
with connection.cursor() as cursor:
sql = """
SELECT total, nombre
FROM totales
WHERE
nombre LIKE :nombre
ORDER BY total DESC
LIMIT 50
"""
cursor.execute(sql, {"nombre": f"{prefijo}%"})
datos = [(r["total"], r["nombre"]) for r in cursor.fetchall()]
else:
with connection.cursor() as cursor:
sql = """
SELECT contador, nombre
FROM nombres
WHERE
anio = :anio AND
nombre LIKE :nombre
ORDER BY contador DESC
LIMIT 50
"""
cursor.execute(sql, {"anio": año, "nombre": f"{prefijo}%"})
datos = [(r["contador"], r["nombre"]) for r in cursor.fetchall()]
if genero:
datos = split_por_genero(datos)[genero]
datos = datos[:10]
chart = pygal.HorizontalBar(height=400, show_legend=False, show_y_labels=True)
chart.x_labels = [nombre.title() for _, nombre in datos[::-1]]
if len(datos) > 1:
chart.title = f"¿Puede ser ... {datos[0][1].title()}? ¿O capaz que {datos[1][1].title()}? ¡Contáme más!"
elif len(datos) == 1:
chart.title = f"¡Hola {datos[0][1].title()}!"
elif len(datos) < 1:
chart.title = "¡No esssistís!"
chart.add("", [contador for contador, _ in datos[::-1]])
return (
chart.render(is_unicode=True),
200,
{"Content-Type": "image/svg+xml"},
)

View File

@@ -1,11 +0,0 @@
from .handler import handle
# Test your handler here
# To disable testing, you can set the build_arg `TEST_ENABLED=false` on the CLI or in your stack.yml
# https://docs.openfaas.com/reference/yaml/#function-build-args-build-args
def test_handle():
# assert handle("input") == "input"
pass

243
busqueda/public/index.html Normal file
View File

@@ -0,0 +1,243 @@
<!DOCTYPE html>
<html lang="en" data-theme="dark">
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1" />
<script
type="text/javascript"
src="https://www.gstatic.com/charts/loader.js"
></script>
<link rel="preconnect" href="https://fonts.googleapis.com" />
<link rel="preconnect" href="https://fonts.gstatic.com" crossorigin />
<!-- Font -->
<link rel="preconnect" href="https://fonts.googleapis.com" />
<link rel="preconnect" href="https://fonts.gstatic.com" crossorigin />
<link
href="https://fonts.googleapis.com/css2?family=Quicksand:wght@300..700&display=swap"
rel="stylesheet"
/>
<link
rel="stylesheet"
href="https://cdn.jsdelivr.net/npm/@picocss/pico@2/css/pico.min.css"
/>
<link
rel="stylesheet"
href="https://cdn.jsdelivr.net/npm/@picocss/pico@2/css/pico.colors.min.css"
/>
<link rel="preconnect" href="https://fonts.googleapis.com" />
<link rel="preconnect" href="https://fonts.gstatic.com" crossorigin />
<link
href="https://fonts.googleapis.com/css2?family=Kode+Mono&display=swap"
rel="stylesheet"
/>
<script type="text/javascript">
google.charts.load("current", { packages: ["corechart"] });
google.charts.setOnLoadCallback(drawChart);
async function drawChart() {
fetch("/", {
method: "POST",
headers: {
"Content-Type": "application/json",
},
body: JSON.stringify({
p: document.getElementById("p").value,
g: document.getElementById("g").value,
a: document.getElementById("a").value,
}),
})
.then((response) => response.json())
.then((json) => {
title = json["title"];
data = json["data"];
console.log(data);
data = google.visualization.arrayToDataTable(data);
var options = {
title: title,
titleTextStyle: {
color: "#aaa",
},
animation: {
startup: true,
duration: 1000,
easing: "out",
},
backgroundColor: "#1c212c",
vAxis: {
minValue: 0,
gridlines: { color: "#666" },
minorGridlines: { color: "#1c212c" },
textStyle: { color: "#aaa" }
},
hAxis: {
gridlines: { color: "#666" },
minorGridlines: { color: "#1c212c" },
textStyle: { color: "#aaa" }
},
legend: { position: 'none'},
};
var chart = new google.visualization.BarChart(
document.getElementById("chart")
);
chart.draw(data, options);
});
}
</script>
<style>
html * {
font-family: "Quicksand", sans-serif;
}
</style>
</head>
<body>
<main class="container" style="text-align: center">
<header>
<h1>Adivinar nombres por edad y género</h1>
</header>
<div id="chart" style="width: 80vw; height: 50vh; margin: auto"></div>
<form
onSubmit="return false;"
style="margin: auto; margin-top: 2em; width: 80%"
>
<fieldset class="grid">
<input type="text" name="p" id="p" placeholder="Como empieza tu nombre?" />
<select name="a" id="a">
<option selected value="">Año?</option>
<option value="1922">1922</option>
<option value="1923">1923</option>
<option value="1924">1924</option>
<option value="1925">1925</option>
<option value="1926">1926</option>
<option value="1927">1927</option>
<option value="1928">1928</option>
<option value="1929">1929</option>
<option value="1930">1930</option>
<option value="1931">1931</option>
<option value="1932">1932</option>
<option value="1933">1933</option>
<option value="1934">1934</option>
<option value="1935">1935</option>
<option value="1936">1936</option>
<option value="1937">1937</option>
<option value="1938">1938</option>
<option value="1939">1939</option>
<option value="1940">1940</option>
<option value="1941">1941</option>
<option value="1942">1942</option>
<option value="1943">1943</option>
<option value="1944">1944</option>
<option value="1945">1945</option>
<option value="1946">1946</option>
<option value="1947">1947</option>
<option value="1948">1948</option>
<option value="1949">1949</option>
<option value="1950">1950</option>
<option value="1951">1951</option>
<option value="1952">1952</option>
<option value="1953">1953</option>
<option value="1954">1954</option>
<option value="1955">1955</option>
<option value="1956">1956</option>
<option value="1957">1957</option>
<option value="1958">1958</option>
<option value="1959">1959</option>
<option value="1960">1960</option>
<option value="1961">1961</option>
<option value="1962">1962</option>
<option value="1963">1963</option>
<option value="1964">1964</option>
<option value="1965">1965</option>
<option value="1966">1966</option>
<option value="1967">1967</option>
<option value="1968">1968</option>
<option value="1969">1969</option>
<option value="1970">1970</option>
<option value="1971">1971</option>
<option value="1972">1972</option>
<option value="1973">1973</option>
<option value="1974">1974</option>
<option value="1975">1975</option>
<option value="1976">1976</option>
<option value="1977">1977</option>
<option value="1978">1978</option>
<option value="1979">1979</option>
<option value="1980">1980</option>
<option value="1981">1981</option>
<option value="1982">1982</option>
<option value="1983">1983</option>
<option value="1984">1984</option>
<option value="1985">1985</option>
<option value="1986">1986</option>
<option value="1987">1987</option>
<option value="1988">1988</option>
<option value="1989">1989</option>
<option value="1990">1990</option>
<option value="1991">1991</option>
<option value="1992">1992</option>
<option value="1993">1993</option>
<option value="1994">1994</option>
<option value="1995">1995</option>
<option value="1996">1996</option>
<option value="1997">1997</option>
<option value="1998">1998</option>
<option value="1999">1999</option>
<option value="2000">2000</option>
<option value="2001">2001</option>
<option value="2002">2002</option>
<option value="2003">2003</option>
<option value="2004">2004</option>
<option value="2005">2005</option>
<option value="1966">1966</option>
<option value="1967">1967</option>
<option value="1968">1968</option>
<option value="1969">1969</option>
<option value="1970">1970</option>
<option value="1971">1971</option>
<option value="1972">1972</option>
<option value="1973">1973</option>
<option value="1974">1974</option>
<option value="1975">1975</option>
<option value="1976">1976</option>
<option value="1977">1977</option>
<option value="1978">1978</option>
<option value="1979">1979</option>
<option value="1980">1980</option>
<option value="1981">1981</option>
<option value="1982">1982</option>
<option value="1983">1983</option>
<option value="1984">1984</option>
<option value="1985">1985</option>
<option value="1986">1986</option>
<option value="1987">1987</option>
<option value="1988">1988</option>
<option value="1989">1989</option>
<option value="1990">1990</option>
<option value="1991">1991</option>
<option value="1992">1992</option>
<option value="1993">1993</option>
<option value="1994">1994</option>
<option value="1995">1995</option>
<option value="1996">1996</option>
<option value="1997">1997</option>
<option value="1998">1998</option>
<option value="1999">1999</option>
<option value="2000">2000</option>
<option value="2001">2001</option>
<option value="2002">2002</option>
<option value="2003">2003</option>
<option value="2004">2004</option>
<option value="2005">2005</option>
</select>
<select name="g" id="g" />
<option selected value="">Género?</option>
<option value="f">Femenino</option>
<option value="m">Masculino</option>
</select>
<input type="submit" value="Buscar" onCLick="drawChart();" />
</fieldset>
</form>
</main>
</body>
</html>

View File

@@ -1,3 +0,0 @@
pygal
requests
pyrqlite

18
busqueda/shard.yml Normal file
View File

@@ -0,0 +1,18 @@
name: busqueda
version: 0.1.0
targets:
funko:
main: main.cr
dependencies:
kemal:
github: kemalcr/kemal
pg:
github: will/crystal-pg
pool:
github: ysbaddaden/pool
# development_dependencies:
# webmock:
# github: manastech/webmock.cr

View File

@@ -1,5 +1,28 @@
#!/bin/sh -x
set -e
#export OPENFAAS_URL=http://pinky:8082
#pass faas.ralsina.me | faas-cli login -u admin --password-stdin
pass faas.ralsina.me | faas-cli login --password-stdin
faas-cli deploy -f nombres.yml
#pass iol-pass | faas-cli secret create iol-pass
#pass iol-user | faas-cli secret create iol-user
#pass iol-api-secret | faas-cli secret create iol-api-secret
#pass nombres-user | faas-cli secret create nombres-user
#pass nombres-pass | faas-cli secret create nombres-pass
#faas-cli deploy -f functions.yml $*
export FAASO_SERVER=http://rocky:8888/admin
pass faaso-rocky | faaso login
pass nombres-user | faaso secret -a historico user
pass nombres-pass | faaso secret -a historico pass
echo "192.168.0.98" | faaso secret -a historico dbhost
faaso build busqueda
faaso scale busqueda 0
faaso scale busqueda 1
pass nombres-user | faaso secret -a busqueda user
pass nombres-pass | faaso secret -a busqueda pass
echo "192.168.0.98" | faaso secret -a busqueda dbhost
faaso build historico
faaso scale historico 0
faaso scale historico 1
rsync -rav nombres.ralsina.me/* ralsina@pinky:/data/websites/nombres.ralsina.me/

18
functions.yml Normal file
View File

@@ -0,0 +1,18 @@
version: 1.0
provider:
name: openfaas
gateway: http://pinky:8082
functions:
tapas:
lang: python3-flask
handler: ./tapas
image: ralsina/tapas:latest
iol:
lang: python3-fastapi
handler: ./iol
image: ralsina/iol:latest
secrets:
- iol-pass
- iol-user
- iol-api-secret

11
historico/README.md Normal file
View File

@@ -0,0 +1,11 @@
# Readme for Historico
This is a funko using the Crystal runtime for [FaaSO](https://git.ralsina.me/ralsina/faaso)
## What is Historico
Write here what it is
## How to use Historico
And so on.

66
historico/funko.cr Normal file
View File

@@ -0,0 +1,66 @@
require "json"
require "kemal"
require "pg"
require "pool/connection"
# get credentials from secrets
USER = File.read("/secrets/user").strip
PASS = File.read("/secrets/pass").strip
DBHOST = File.read("/secrets/dbhost").strip
DBURL = "postgres://#{USER}:#{PASS}@#{DBHOST}:5432/nombres"
puts "Connnecting to #{DBURL}"
# Create a connection pool to the database
pg = ConnectionPool.new(capacity: 5, timeout: 1.seconds) do
PG.connect(DBURL)
end
# Connect to the database and get information about
# the requested names
get "/" do |env|
# Names are query parameters
# Split by commas, capitalize and take the first 5
names = env.params.query["names"]
.split(",").map(&.strip.capitalize)[..4]
# Prepare results table
results = [] of Array(String)
results << ["Año"] + names
(1922..2015).each do |anio|
results << [anio.to_s]
end
# Connect using credentials provided
pg.connection do |cursor|
# Get the information for each name
names.map do |name|
# Normalize: remove diacritics etc.
name = name.unicode_normalize(:nfkd)
.chars.reject! { |character|
!character.ascii_letter? && (character != ' ')
}.join("").downcase
counter_per_year = {} of Int32 => Int32
cursor.query("
SELECT anio::integer, contador::integer
FROM nombres WHERE nombre = $1", name) do |result_set|
result_set.each do
counter_per_year[result_set.read(Int32)] = result_set.read(Int32)
end
end
(1922..2015).each do |anio|
results[anio - 1921] << counter_per_year.fetch(anio, 0).to_s
end
end
end
results.to_json
end
# The `/ping/` endpoint is configured in the container as a healthcheck
# You can make it better by checking that your database is responding
# or whatever checks you think are important
#
get "/ping/" do
pg.connection.exec("SELECT 42")
"OK"
end

11
historico/funko.yml Normal file
View File

@@ -0,0 +1,11 @@
name: historico
runtime: kemal
options:
shard_build_options: "--release -d --error-trace"
ship_packages: []
devel_packages: []
healthcheck_options: "--interval=1m --timeout=2s --start-period=2s --retries=3"
healthcheck_command: "curl --fail http://localhost:3000/ping || exit 1"
copy_from_build:
- "public public"
- "bin/funko ."

View File

@@ -1,66 +0,0 @@
import unicodedata
import urllib
from collections import defaultdict as ddict
from dataclasses import dataclass
from json import loads
import pygal
import pyrqlite.dbapi2 as dbapi2
import requests
connection = dbapi2.connect(
host="10.61.0.1",
user="root",
port=4001,
password="",
)
def remove_accents(input_str):
nfkd_form = unicodedata.normalize("NFKD", input_str)
return "".join([c for c in nfkd_form if not unicodedata.combining(c)])
def handle(req):
"""handle a request to the function
Args:
req (str): request body
{"i": ["nombre1, nombre2"]}
"""
nombres = []
try:
nombres = loads(req)
nombres = nombres["i"].split(",")
nombres = [remove_accents(x.strip().lower()) for x in nombres]
nombres = [n for n in nombres if n]
except Exception:
pass
if not nombres:
nombres = ["maria", "juan"]
chart = pygal.Line(
height=200, fill=True, human_readable=True, show_minor_x_labels=False
)
chart.x_labels = [str(x) for x in range(1922, 2015)]
chart.x_labels_major = [str(x) if x % 10 == 0 else "" for x in range(1922, 2015)]
for nombre in nombres:
datos = ddict(int)
with connection.cursor() as cursor:
sql = """
SELECT anio, contador, nombre
FROM nombres
WHERE nombre = :nombre
ORDER BY anio
"""
cursor.execute(sql, {"nombre": nombre})
datos.update({r["anio"]: r["contador"] for r in cursor.fetchall()})
chart.add(nombre.title(), [datos[x] for x in range(1922, 2015)])
chart.x_labels = [str(n) for n in range(1922, 2015)]
chart.x_labels_major = [str(n) for n in range(1920, 2020, 10)]
# return Response(chart.render(is_unicode=True), mimetype="image/svg+xml")
return chart.render(is_unicode=True), 200, {"Content-Type": "image/svg+xml"}

109
historico/public/index.html Normal file
View File

@@ -0,0 +1,109 @@
<!DOCTYPE html>
<html lang="en" data-theme="dark">
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1" />
<script
type="text/javascript"
src="https://www.gstatic.com/charts/loader.js"
></script>
<link rel="preconnect" href="https://fonts.googleapis.com" />
<link rel="preconnect" href="https://fonts.gstatic.com" crossorigin />
<!-- Font -->
<link rel="preconnect" href="https://fonts.googleapis.com">
<link rel="preconnect" href="https://fonts.gstatic.com" crossorigin>
<link href="https://fonts.googleapis.com/css2?family=Quicksand:wght@300..700&display=swap"
rel="stylesheet">
<link
rel="stylesheet"
href="https://cdn.jsdelivr.net/npm/@picocss/pico@2/css/pico.min.css"
/>
<link
rel="stylesheet"
href="https://cdn.jsdelivr.net/npm/@picocss/pico@2/css/pico.colors.min.css"
/>
<link rel="preconnect" href="https://fonts.googleapis.com" />
<link rel="preconnect" href="https://fonts.gstatic.com" crossorigin />
<link
href="https://fonts.googleapis.com/css2?family=Kode+Mono&display=swap"
rel="stylesheet"
/>
<script type="text/javascript">
google.charts.load("current", { packages: ["corechart"] });
google.charts.setOnLoadCallback(drawChart);
async function drawChart() {
fetch(
"/faaso/historico/?" +
new URLSearchParams({
names: document.getElementById("nombres").value,
})
)
.then((response) => response.json())
.then((json) => {
var data = [json[0]];
data.push(
...json
.slice(1)
.map((item) => item.map((value) => parseInt(value)))
);
data = google.visualization.arrayToDataTable(data);
var options = {
title: "",
animation: {
startup: true,
duration: 1000,
easing: "out",
},
backgroundColor: "#1c212c",
vAxis: {
minValue: 0,
gridlines: { color: "#666" },
minorGridlines: { color: "#1c212c" },
textStyle: { color: "#aaa" }
},
hAxis: {
gridlines: { color: "#666" },
minorGridlines: { color: "#1c212c" },
textStyle: { color: "#aaa" }
},
legend: { position: "bottom", textStyle: { color: "#aaa" } },
};
var chart = new google.visualization.LineChart(
document.getElementById("chart")
);
chart.draw(data, options);
});
}
</script>
<style>
html * {
font-family: 'Quicksand', sans-serif;
}
</style>
</head>
<body>
<main class="container" style="text-align: center">
<header>
<h1>Popularidad de Nombres en Argentina</h1>
</header>
<div id="chart" style="width: 80vw; height: 50vh; margin: auto"></div>
<form
role="search"
onSubmit="return false;"
style="margin: auto; margin-top: 2em; width: 80%"
>
<input
type="search"
name="nombres"
id="nombres"
placeholder="Nombres separados con comas"
aria-label="Search"
/>
<input type="submit" value="Buscar" onCLick="drawChart();" />
</form>
</main>
</body>
</html>

View File

@@ -1,3 +0,0 @@
pygal
requests
pyrqlite

14
historico/shard.yml Normal file
View File

@@ -0,0 +1,14 @@
name: historico
version: 0.1.0
targets:
funko:
main: main.cr
dependencies:
kemal:
github: kemalcr/kemal
pg:
github: will/crystal-pg
pool:
github: ysbaddaden/pool

View File

@@ -1,41 +0,0 @@
# If you would like to disable
# automated testing during faas-cli build,
# Replace the content of this file with
# [tox]
# skipsdist = true
# You can also edit, remove, or add additional test steps
# by editing, removing, or adding new testenv sections
# find out more about tox: https://tox.readthedocs.io/en/latest/
[tox]
envlist = lint,test
skipsdist = true
[testenv:test]
deps =
flask
pytest
-rrequirements.txt
commands =
# run unit tests with pytest
# https://docs.pytest.org/en/stable/
# configure by adding a pytest.ini to your handler
pytest
[testenv:lint]
deps =
flake8
commands =
flake8 .
[flake8]
count = true
max-line-length = 127
max-complexity = 10
statistics = true
# stop the build if there are Python syntax errors or undefined names
select = E9,F63,F7,F82
show-source = true

157
iol/handler.py Normal file
View File

@@ -0,0 +1,157 @@
import logging
import time
from functools import lru_cache
import requests as r
from fastapi import APIRouter
BASE_URL = "https://api.invertironline.com/"
USER = open("/var/openfaas/secrets/iol-user").read().strip()
PASS = open("/var/openfaas/secrets/iol-pass").read().strip()
SECRET = open("/var/openfaas/secrets/iol-api-secret").read().strip()
router = APIRouter()
last_results = {}
def get_ttl_hash(seconds=3600):
"""Return the same value withing `seconds` time period"""
return round(time.time() / seconds)
@lru_cache
def get_token(ttl_hash=None):
logging.error("getting token")
url = BASE_URL + "token"
data = {"username": USER, "password": PASS, "grant_type": "password"}
response = r.post(url, data=data)
access_token = response.json()["access_token"]
refresh_token = response.json()["refresh_token"]
return access_token, refresh_token
def refresh_token(refresh_token):
url = BASE_URL + "token"
data = {"refresh_token": refresh_token, "grant_type": "refresh_token"}
response = r.post(url, data=data)
access_token = response.json()["access_token"]
refresh_token = response.json()["refresh_token"]
return access_token, refresh_token
paises = {
"US": "estados_Unidos",
"AR": "argentina",
}
@lru_cache
def get(url, ttl_hash=None):
logging.error("getting data")
access, refresh = get_token(ttl_hash=get_ttl_hash())
response = r.get(
url, headers={"Authorization": "Bearer " + access, "Accept": "application/json"}
)
return response.json()
@router.get("/{secret}/accion/{pais}/{accion}")
def get_accion(secret, pais, accion):
if secret != SECRET:
raise Exception("BAD")
pais = pais.upper()
accion = accion.upper()
try:
access, refresh = get_token(ttl_hash=get_ttl_hash())
url = (
BASE_URL
+ f"/api/v2/Cotizaciones/acciones/{pais}/Todos?cotizacionInstrumentoModel.instrumento=acciones&cotizacionInstrumentoModel.pais={paises[pais]}&api_key={access}"
)
data = get(url, ttl_hash=get_ttl_hash())
last_results[f"accion/{pais}/{accion}"] = [
a for a in data["titulos"] if a["simbolo"] == accion
][0]
except:
pass
return last_results[f"accion/{pais}/{accion}"]
@router.get("/{secret}/bono/{nombre}")
def get_bono(secret, nombre):
if secret != SECRET:
raise Exception("BAD")
nombre = nombre.upper()
access, refresh = get_token(ttl_hash=get_ttl_hash())
try:
url = (
BASE_URL
+ f"/api/v2/Cotizaciones/titulosPublicos/ar/Todos?cotizacionInstrumentoModel.instrumento=titulosPublicos&cotizacionInstrumentoModel.pais=argentina&api_key={access}"
)
data = get(url, ttl_hash=get_ttl_hash())
last_results[f"bono/{nombre}"] = [
a for a in data["titulos"] if a["simbolo"] == nombre
][0]
except:
pass
return last_results[f"bono/{nombre}"]
@router.get("/{secret}/cedear/{nombre}")
def get_cedear(secret, nombre):
if secret != SECRET:
raise Exception("BAD")
nombre = nombre.upper()
access, refresh = get_token(ttl_hash=get_ttl_hash())
try:
url = (
BASE_URL
+ f"/api/v2/Cotizaciones/cedears/ar/Todos?cotizacionInstrumentoModel.instrumento=cedears&cotizacionInstrumentoModel.pais=argentina&api_key={access}"
)
data = get(url, ttl_hash=get_ttl_hash())
last_results[f"cedear/{nombre}"] = [
a for a in data["titulos"] if a["simbolo"] == nombre
][0]
except:
pass
return last_results[f"cedear/{nombre}"]
@router.get("/{secret}/ON/{nombre}")
def get_on(secret, nombre):
if secret != SECRET:
raise Exception("BAD")
nombre = nombre.upper()
access, refresh = get_token(ttl_hash=get_ttl_hash())
try:
url = (
BASE_URL
+ f"/api/v2/Cotizaciones/obligacionesnegociables/ar/Todos?cotizacionInstrumentoModel.instrumento=obligacionesNegociables&cotizacionInstrumentoModel.pais=argentina&api_key={access}"
)
data = get(url, ttl_hash=get_ttl_hash())
last_results[f"ON/{nombre}"] = [
a for a in data["titulos"] if a["simbolo"] == nombre
][0]
except:
pass
return last_results[f"ON/{nombre}"]
@router.get("/{secret}/ADR/{nombre}")
def get_adrs(secret, nombre):
if secret != SECRET:
raise Exception("BAD")
nombre = nombre.upper()
access, refresh = get_token(ttl_hash=get_ttl_hash())
try:
url = (
BASE_URL
+ f"/api/v2/Cotizaciones/adrs/us/Todos?cotizacionInstrumentoModel.instrumento=aDRs&cotizacionInstrumentoModel.pais=estados_Unidos&api_key={access}"
)
data = get(url, ttl_hash=get_ttl_hash())
last_results[f"ADR/{nombre}"] = [
a for a in data["titulos"] if a["simbolo"] == nombre
][0]
except:
pass
return last_results[f"ADR/{nombre}"]

3
iol/requirements.txt Normal file
View File

@@ -0,0 +1,3 @@
fastapi
requests
uvicorn[standard]

View File

@@ -0,0 +1,339 @@
<!DOCTYPE html>
<html lang="en" data-theme="dark">
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1" />
<script
type="text/javascript"
src="https://www.gstatic.com/charts/loader.js"
></script>
<link rel="preconnect" href="https://fonts.googleapis.com" />
<link rel="preconnect" href="https://fonts.gstatic.com" crossorigin />
<!-- Font -->
<link rel="preconnect" href="https://fonts.googleapis.com" />
<link rel="preconnect" href="https://fonts.gstatic.com" crossorigin />
<link
href="https://fonts.googleapis.com/css2?family=Quicksand:wght@300..700&display=swap"
rel="stylesheet"
/>
<link
rel="stylesheet"
href="https://cdn.jsdelivr.net/npm/@picocss/pico@2/css/pico.min.css"
/>
<link
rel="stylesheet"
href="https://cdn.jsdelivr.net/npm/@picocss/pico@2/css/pico.colors.min.css"
/>
<link rel="preconnect" href="https://fonts.googleapis.com" />
<link rel="preconnect" href="https://fonts.gstatic.com" crossorigin />
<link
href="https://fonts.googleapis.com/css2?family=Kode+Mono&display=swap"
rel="stylesheet"
/>
<script type="text/javascript">
google.charts.load("current", { packages: ["corechart"] });
google.charts.setOnLoadCallback(drawChart);
async function drawChart() {
drawChart1();
drawChart2();
}
async function drawChart1() {
fetch("https://faaso-prod.ralsina.me/faaso/busqueda/", {
method: "POST",
headers: {
"Content-Type": "application/json",
},
body: JSON.stringify({
p: document.getElementById("p").value,
g: document.getElementById("g").value,
a: document.getElementById("a").value,
}),
})
.then((response) => response.json())
.then((json) => {
title = json["title"];
data = json["data"];
console.log(data);
data = google.visualization.arrayToDataTable(data);
var options = {
title: title,
titleTextStyle: {
color: "#aaa",
},
animation: {
startup: true,
duration: 1000,
easing: "out",
},
backgroundColor: "#1c212c",
vAxis: {
minValue: 0,
gridlines: { color: "#666" },
minorGridlines: { color: "#1c212c" },
textStyle: { color: "#aaa" }
},
hAxis: {
gridlines: { color: "#666" },
minorGridlines: { color: "#1c212c" },
textStyle: { color: "#aaa" }
},
legend: { position: 'none'},
};
var chart1 = new google.visualization.BarChart(
document.getElementById("chart1")
);
chart1.draw(data, options);
});
}
async function drawChart2() {
fetch(
"https://faaso-prod.ralsina.me/faaso/historico/?" +
new URLSearchParams({
names: document.getElementById("nombres").value,
})
)
.then((response) => response.json())
.then((json) => {
var data = [json[0]];
data.push(
...json
.slice(1)
.map((item) => item.map((value) => parseInt(value)))
);
data = google.visualization.arrayToDataTable(data);
var options = {
title: "",
animation: {
startup: true,
duration: 1000,
easing: "out",
},
backgroundColor: "#1c212c",
vAxis: {
minValue: 0,
gridlines: { color: "#666" },
minorGridlines: { color: "#1c212c" },
textStyle: { color: "#aaa" }
},
hAxis: {
gridlines: { color: "#666" },
minorGridlines: { color: "#1c212c" },
textStyle: { color: "#aaa" }
},
legend: { position: "bottom", textStyle: { color: "#aaa" } },
};
var chart2 = new google.visualization.LineChart(
document.getElementById("chart2")
);
chart2.draw(data, options);
});
}
</script>
<style>
html * {
font-family: "Quicksand", sans-serif;
}
</style>
</head>
<body>
<main class="container" style="text-align: center">
<header>
<h1>Cosas con Datos de Nombres de Argentina</h1>
<h2>Adivinar nombres por edad y género</h2>
</header>
<div id="chart1" style="width: 80vw; height: 50vh; margin: auto"></div>
<form
onSubmit="return false;"
style="margin: auto; margin-top: 2em; width: 80%"
>
<fieldset class="grid">
<input type="text" name="p" id="p" placeholder="Como empieza tu nombre?" />
<select name="a" id="a">
<option selected value="">Año?</option>
<option value="1922">1922</option>
<option value="1923">1923</option>
<option value="1924">1924</option>
<option value="1925">1925</option>
<option value="1926">1926</option>
<option value="1927">1927</option>
<option value="1928">1928</option>
<option value="1929">1929</option>
<option value="1930">1930</option>
<option value="1931">1931</option>
<option value="1932">1932</option>
<option value="1933">1933</option>
<option value="1934">1934</option>
<option value="1935">1935</option>
<option value="1936">1936</option>
<option value="1937">1937</option>
<option value="1938">1938</option>
<option value="1939">1939</option>
<option value="1940">1940</option>
<option value="1941">1941</option>
<option value="1942">1942</option>
<option value="1943">1943</option>
<option value="1944">1944</option>
<option value="1945">1945</option>
<option value="1946">1946</option>
<option value="1947">1947</option>
<option value="1948">1948</option>
<option value="1949">1949</option>
<option value="1950">1950</option>
<option value="1951">1951</option>
<option value="1952">1952</option>
<option value="1953">1953</option>
<option value="1954">1954</option>
<option value="1955">1955</option>
<option value="1956">1956</option>
<option value="1957">1957</option>
<option value="1958">1958</option>
<option value="1959">1959</option>
<option value="1960">1960</option>
<option value="1961">1961</option>
<option value="1962">1962</option>
<option value="1963">1963</option>
<option value="1964">1964</option>
<option value="1965">1965</option>
<option value="1966">1966</option>
<option value="1967">1967</option>
<option value="1968">1968</option>
<option value="1969">1969</option>
<option value="1970">1970</option>
<option value="1971">1971</option>
<option value="1972">1972</option>
<option value="1973">1973</option>
<option value="1974">1974</option>
<option value="1975">1975</option>
<option value="1976">1976</option>
<option value="1977">1977</option>
<option value="1978">1978</option>
<option value="1979">1979</option>
<option value="1980">1980</option>
<option value="1981">1981</option>
<option value="1982">1982</option>
<option value="1983">1983</option>
<option value="1984">1984</option>
<option value="1985">1985</option>
<option value="1986">1986</option>
<option value="1987">1987</option>
<option value="1988">1988</option>
<option value="1989">1989</option>
<option value="1990">1990</option>
<option value="1991">1991</option>
<option value="1992">1992</option>
<option value="1993">1993</option>
<option value="1994">1994</option>
<option value="1995">1995</option>
<option value="1996">1996</option>
<option value="1997">1997</option>
<option value="1998">1998</option>
<option value="1999">1999</option>
<option value="2000">2000</option>
<option value="2001">2001</option>
<option value="2002">2002</option>
<option value="2003">2003</option>
<option value="2004">2004</option>
<option value="2005">2005</option>
<option value="1966">1966</option>
<option value="1967">1967</option>
<option value="1968">1968</option>
<option value="1969">1969</option>
<option value="1970">1970</option>
<option value="1971">1971</option>
<option value="1972">1972</option>
<option value="1973">1973</option>
<option value="1974">1974</option>
<option value="1975">1975</option>
<option value="1976">1976</option>
<option value="1977">1977</option>
<option value="1978">1978</option>
<option value="1979">1979</option>
<option value="1980">1980</option>
<option value="1981">1981</option>
<option value="1982">1982</option>
<option value="1983">1983</option>
<option value="1984">1984</option>
<option value="1985">1985</option>
<option value="1986">1986</option>
<option value="1987">1987</option>
<option value="1988">1988</option>
<option value="1989">1989</option>
<option value="1990">1990</option>
<option value="1991">1991</option>
<option value="1992">1992</option>
<option value="1993">1993</option>
<option value="1994">1994</option>
<option value="1995">1995</option>
<option value="1996">1996</option>
<option value="1997">1997</option>
<option value="1998">1998</option>
<option value="1999">1999</option>
<option value="2000">2000</option>
<option value="2001">2001</option>
<option value="2002">2002</option>
<option value="2003">2003</option>
<option value="2004">2004</option>
<option value="2005">2005</option>
</select>
<select name="g" id="g" />
<option selected value="">Género?</option>
<option value="f">Femenino</option>
<option value="m">Masculino</option>
</select>
<input type="submit" value="Buscar" onCLick="drawChart1();" />
</fieldset>
</form>
</main>
<main class="container" style="text-align: center">
<header>
<h2>Popularidad de Nombres en Argentina</h2>
</header>
<div id="chart2" style="width: 80vw; height: 50vh; margin: auto"></div>
<form
role="search"
onSubmit="return false;"
style="margin: auto; margin-top: 2em; width: 80%"
>
<input
type="search"
name="nombres"
id="nombres"
placeholder="Nombres separados con comas"
aria-label="Search"
/>
<input type="submit" value="Buscar" onCLick="drawChart2();" />
</form>
</main>
<main class="container">
<header>
<h2>Cosas Nerds</h2>
</header>
<ul>
<li>Los datos utilizados son provistos por el RENAPER (Registro Nacional de las Personas), y sólo son útiles para Argentina: <a href="http://www.datos.gob.ar/dataset/otros-nombres-personas-fisicas">Los Datos</a>
<li>Le saqué todos los acentos a la data. Después de todo, si te llamás María y te digo Maria no te vas a ofender.
<li>Había una página del RENAPER que hacía algo parecido pero era una porquería. Ahora hay otra que ni miré.
<li>La data no es perfecta. En el caso de algunos nombres muy peculiares están duplicados. Por ejemplo, mi cuñado se llama "Pedro Fuat", y hay dos registros en el mismo año. Sospecho que es un duplicado de él mismo. Nota de color: los tres hijos de mi suegra tienen nombres que nadie más tiene en la historia de la Argentina.
<li>Los datos están publicados de manera conveniente <a href="https://www.dolthub.com/repositories/ralsina/nombres_argentina_1922_2005/doc/main">en dolthub.</a>
<li>La detección de género está hecha en base a datos del INE de España (lista de los nombres de personas nacidas en España con más de 20 ocurrencias)
<li>Si alguien tiene data similar de otros países me encantaría hacer páginas similares.
<li>Si les interesa es muy fácil hacer un Jupyter Notebook para jugar con estos datos.
<li>Código y cosas: https://github.com/ralsina/nombres
</ul>
La aplicación consiste de:
<ul>
<li>Una página estática (ésta!)
<li>Los charts SVG están hechos con Google Charts
<li>El código de backend está hecho en Crystal
<li>Todo está ejecutándose en una computadora categoría RaspBerry Pi usando <a href="faaso.ralsina.me">faaso</a> con una base de datos Postgres
</ul>
</main>
</body>
</html>

View File

@@ -1,17 +0,0 @@
version: 1.0
provider:
name: openfaas
gateway: http://pinky:8082
functions:
busqueda:
lang: python3-flask
handler: ./busqueda
image: ralsina/nombres_busqueda:latest
historico:
lang: python3-flask
handler: ./historico
image: ralsina/nombres_historico:latest
tapas:
lang: python3-flask
handler: ./tapas
image: ralsina/tapas:latest

228
old-busqueda/handler.cr Normal file
View File

@@ -0,0 +1,228 @@
require "http/client"
require "http/headers"
require "http/request"
require "ishi/html"
require "json"
require "uuid"
require "db"
require "pg"
USER = File.read("/var/openfaas/secrets/nombres-user").strip
PASS = File.read("/var/openfaas/secrets/nombres-pass").strip
DB_URL = "postgres://#{USER}:#{PASS}@10.61.0.1:5432/nombres"
class Handler
# This class is the entry point for the OpenFaaS function.
# run() is the important bit
def format_buffer(buffer, canvas_name, title = "")
# Process the gnuplot output so it works in the page
#
# buffer is the Ishi output
# name is a string to replace for gnuplot_canvas so
# we can have multiple charts in a page
# title is added on top of the chart
html = buffer.to_s.split("\n")
html = html[html.index("<script type=\"text/javascript\">")..html.index("</script>")]
html = "<b>#{title}</b>" + html.join("\n") + %(
<div class="gnuplot">
<canvas id="Tile" width="32" height="32" hidden></canvas>
<table class="plot">
<tr><td>
<canvas id="gnuplot_canvas" width="800" height="300" tabindex="0">
Sorry, your browser seems not to support the HTML 5 canvas element
</canvas>
</td></tr>
</table>
<script type="text/javascript" defer>
gnuplot.init(); gnuplot_canvas();
</script>
</div>
)
# This ID needs to be unique in case
# we have 2 charts in the same page
html.gsub("gnuplot_canvas", canvas_name)
end
def normalize_name(s)
# Remove diacritics, turn lowercase
normalized = s.unicode_normalize(:nfkd).chars
normalized.reject! { |character|
!character.ascii_letter?
}.join("").downcase
end
def run(request : HTTP::Request)
# Try to find most popular names based on a prefix, year and gender.
#
# Request body is JSON in this form:
#
# {
# p: prefijo del nombre,
# g: genero del nombre,
# y: year de nacimiento
# }
if (body = request.body).nil?
query = {"p": "", "g": "", "a": ""}
else
query = Hash(String, String).from_json(body)
end
# Sanitize input.
# Each one either a valid string or nil
prefijo = query.fetch("p", "")
genero = query.fetch("g", "")
year = query.fetch("y", "")
if !prefijo.empty?
prefijo = normalize_name(prefijo)
else
prefijo = nil
end
if !["f", "m"].includes?(genero)
genero = nil
end
year = year.to_i?
datos = [] of Tuple(Int32, String)
DB.open(DB_URL) do |cursor|
if prefijo.nil? && year.nil?
# Global totals
result_set = cursor.query("
SELECT total::integer, nombre
FROM totales
ORDER BY total DESC
LIMIT 50")
elsif prefijo.nil? && !year.nil?
# Per-year totals
result_set = cursor.query("
SELECT contador::integer, nombre
FROM nombres
WHERE
anio = $1
ORDER BY contador DESC
LIMIT 50", year)
elsif !prefijo.nil? && year.nil?
# Filter only by prefix
result_set = cursor.query("
SELECT total::integer, nombre
FROM totales
WHERE
nombre LIKE $1
ORDER BY total DESC
LIMIT 50", prefijo + "%")
elsif !prefijo.nil? && !year.nil?
# We have both
result_set = cursor.query("
SELECT contador::integer, nombre
FROM nombres
WHERE
anio = $1 AND
nombre LIKE $2
ORDER BY contador DESC
LIMIT 50", year, prefijo + "%")
end
if !result_set.nil?
result_set.each do
valor = result_set.read(Int32)
nombre = result_set.read(String)
datos.push({valor, nombre})
end
result_set.close
end
end
puts "Data gathered"
if datos.empty?
# This is bad 😀
return {
body: "Que raro, no tengo *idea*!",
status_code: 200,
headers: HTTP::Headers{"Content-Type" => "text/html"},
}
end
# In this context, remove all composite names
datos.reject! { |row|
row[1].to_s.includes? " "
}
if genero
DB.open(DB_URL) do |cursor|
datos.reject! { |row|
# How feminine is this name?
# Yes this database is upper case
puts "Checking #{row[0]} #{row[1]}"
feminidad = 0
sql = %(
SELECT COALESCE((SELECT frecuencia FROM mujeres WHERE nombre='#{row[1]?.to_s.upcase}'), 0) AS mujeres,
COALESCE((SELECT frecuencia FROM hombres WHERE nombre='#{row[1]?.to_s.upcase}'), 0) AS hombres
)
puts "SQL: #{sql}"
cursor.query sql do |result_set|
result_set.each do
mujeres = result_set.read(Int32)
hombres = result_set.read(Int32)
puts "frecuencias: #{mujeres} #{hombres}"
if hombres == mujeres == 0
feminidad = 0.5
else
feminidad = mujeres / (hombres + mujeres)
end
end
end
# El overlap en 0.5 es intencional!
if (feminidad >= 0.5 && genero == "f") ||
(feminidad <= 0.5 && genero == "m")
false
else
true
end
}
puts "Data split by gender"
end
end
datos = datos[..10]
if datos.size > 1
title = "¿Puede ser ... #{datos[0][1].to_s.titleize}? ¿O capaz que #{datos[1][1].to_s.titleize}? ¡Contame más!"
elsif datos.size == 1
title = "Me parece que ... #{datos[0][1].to_s.titleize}!"
else
title = "No tengo idea!"
end
buffer = IO::Memory.new
Ishi.new(buffer) do
x = (0..datos.size - 1).to_a
y = datos.map { |row|
row[0].to_f / 1000
}
yrange(0..(y.max*1.1).to_i + 1)
xtics = Hash(Float64, String).new
datos.each_with_index { |row, i|
xtics[i.to_f] = row[1].to_s.titleize
}
canvas_size(800, 300)
plot(x, y, style: :boxes, fs: 0.25)
.boxwidth(0.5)
.show_key(false)
.ylabel("Popularidad (miles)")
.xtics(xtics)
end
{
body: format_buffer(buffer, "busqueda", title),
status_code: 200,
headers: HTTP::Headers{"Content-Type" => "text/html"},
}
end
end

8
old-busqueda/shard.yml Normal file
View File

@@ -0,0 +1,8 @@
name: busqueda
version: 0.1.0
dependencies:
ishi:
github: toddsundsted/ishi
pg:
github: will/crystal-pg

View File

@@ -15,6 +15,8 @@ def handle(req):
"author": "bat",
}
"""
if not req:
return "Foo", 200, {"Content-Type": "text/plain"}
try:
args = loads(req)
except Exception:
@@ -24,9 +26,8 @@ def handle(req):
byte_arr = BytesIO()
c.image.save(byte_arr, format="JPEG")
# return Response(chart.render(is_unicode=True), mimetype="image/svg+xml")
return (
f'<img src="data:image/jpeg;base64, {base64.b64encode(byte_arr.getvalue()).decode("utf-8")}">',
f'<img src="data:image/jpeg;base64, {base64.b64encode(byte_arr.getvalue()).decode("utf-8")}" style="width: 100%; border: solid 1px #aaa;">',
200,
{"Content-Type": "text/html"},
)

View File

@@ -1,11 +0,0 @@
from .handler import handle
# Test your handler here
# To disable testing, you can set the build_arg `TEST_ENABLED=false` on the CLI or in your stack.yml
# https://docs.openfaas.com/reference/yaml/#function-build-args-build-args
def test_handle():
# assert handle("input") == "input"
pass

View File

@@ -1,41 +0,0 @@
# If you would like to disable
# automated testing during faas-cli build,
# Replace the content of this file with
# [tox]
# skipsdist = true
# You can also edit, remove, or add additional test steps
# by editing, removing, or adding new testenv sections
# find out more about tox: https://tox.readthedocs.io/en/latest/
[tox]
envlist = lint,test
skipsdist = true
[testenv:test]
deps =
flask
pytest
-rrequirements.txt
commands =
# run unit tests with pytest
# https://docs.pytest.org/en/stable/
# configure by adding a pytest.ini to your handler
pytest
[testenv:lint]
deps =
flake8
commands =
flake8 .
[flake8]
count = true
max-line-length = 127
max-complexity = 10
statistics = true
# stop the build if there are Python syntax errors or undefined names
select = E9,F63,F7,F82
show-source = true

View File

@@ -0,0 +1,39 @@
FROM --platform=${TARGETPLATFORM:-linux/amd64} ghcr.io/openfaas/of-watchdog:0.9.10 as watchdog
FROM --platform=${TARGETPLATFORM:-linux/amd64} alpine as build
ARG ADDITIONAL_PACKAGE
COPY --from=watchdog /fwatchdog /usr/bin/fwatchdog
RUN apk update && apk upgrade && apk add crystal shards openssl-dev ${ADDITIONAL_PACKAGE} && apk cache clean
WORKDIR /home/app
COPY function/shard.yml shard.yml
RUN shards install
COPY . .
RUN crystal build main.cr -o handler --error-trace -p && rm -rf /root/.cache
FROM --platform=${TARGETPLATFORM:-linux/amd64} alpine as ship
ARG ADDITIONAL_PACKAGE
RUN apk update && apk upgrade && apk add openssl pcre2 libgcc gc libevent ${ADDITIONAL_PACKAGE} && apk cache clean
# Add non root user
# Add non root user
RUN addgroup -S app && adduser app -S -G app
WORKDIR /home/app
USER app
COPY --from=build /home/app/function/ .
COPY --from=build /home/app/handler .
COPY --from=watchdog /fwatchdog /usr/bin/fwatchdog
ENV fprocess="./handler"
ENV exec_timeout=30
EXPOSE 8080
HEALTHCHECK --interval=2s CMD [ -e /tmp/.lock ] || exit 1
ENV upstream_url="http://127.0.0.1:5000"
ENV mode="http"
CMD ["fwatchdog"]

View File

@@ -0,0 +1,12 @@
require "http/request"
require "http/headers"
class Handler
def run(request : HTTP::Request)
{
body: "Hello, Crystal. You said: #{request.body.try(&.gets_to_end)}",
status_code: 200,
headers: HTTP::Headers{"Content-Type" => "text/plain"},
}
end
end

View File

@@ -0,0 +1,2 @@
name: crystal-http-template
version: 0.1.0

View File

@@ -0,0 +1,41 @@
require "http/server"
require "./function/handler"
server = HTTP::Server.new do |context|
response_triple : NamedTuple(body: String, headers: HTTP::Headers, status_code: Int32) |
NamedTuple(body: String, headers: HTTP::Headers) |
NamedTuple(body: String, status_code: Int32) |
NamedTuple(body: String) |
NamedTuple(headers: HTTP::Headers, status_code: Int32) |
NamedTuple(headers: HTTP::Headers) |
NamedTuple(status_code: Int32)
handler = Handler.new
response_triple = handler.run(context.request)
if response_triple.is_a?(NamedTuple(body: String, headers: HTTP::Headers, status_code: Int32) |
NamedTuple(body: String, status_code: Int32) |
NamedTuple(headers: HTTP::Headers, status_code: Int32) |
NamedTuple(status_code: Int32))
context.response.status_code = response_triple[:status_code]
end
if response_triple.is_a?(NamedTuple(body: String, headers: HTTP::Headers, status_code: Int32) |
NamedTuple(body: String, headers: HTTP::Headers) |
NamedTuple(headers: HTTP::Headers, status_code: Int32) |
NamedTuple(headers: HTTP::Headers))
response_triple[:headers].each do |key, value|
context.response.headers[key] = value
end
end
if response_triple.is_a?(NamedTuple(body: String, headers: HTTP::Headers, status_code: Int32) |
NamedTuple(body: String, headers: HTTP::Headers) |
NamedTuple(body: String, status_code: Int32) |
NamedTuple(body: String))
context.response.print(response_triple[:body])
end
end
server.bind_tcp "0.0.0.0", 5000
server.listen

View File

@@ -0,0 +1,6 @@
language: crystal
fprocess: ./handler
welcome_message: |
You have created a new function which uses crystal 1.0.0.
To include third-party dependencies, use a vendoring tool like shards:
shards documentation: https://github.com/crystal-lang/shards

View File

@@ -0,0 +1,49 @@
FROM --platform=${TARGETPLATFORM:-linux/amd64} ghcr.io/openfaas/of-watchdog:0.9.10 as watchdog
FROM --platform=${TARGETPLATFORM:-linux/amd64} python:3.11-alpine
COPY --from=watchdog /fwatchdog /usr/bin/fwatchdog
RUN chmod +x /usr/bin/fwatchdog
ARG ADDITIONAL_PACKAGE
RUN apk --no-cache add musl-dev gcc make ${ADDITIONAL_PACKAGE}
# Add non root user
RUN addgroup -S app && adduser app -S -G app
RUN chown app /home/app
USER app
ENV PATH=$PATH:/home/app/.local/bin
WORKDIR /home/app/
COPY requirements.txt .
USER root
RUN pip install -r requirements.txt
USER app
COPY index.py .
RUN mkdir -p function
RUN touch ./function/__init__.py
WORKDIR /home/app/function/
COPY function/requirements.txt .
RUN pip install --user -r requirements.txt
WORKDIR /home/app/
USER root
# remove build dependencies
RUN apk del musl-dev gcc make
COPY function function
RUN chown -R app:app ./
USER app
ENV fprocess="uvicorn index:app --workers 1 --host 0.0.0.0 --port 8000"
ENV cgi_headers="true"
ENV mode="http"
ENV upstream_url="http://127.0.0.1:8000"
HEALTHCHECK --interval=5s CMD [ -e /tmp/.lock ] || exit 1
CMD ["fwatchdog"]

View File

@@ -0,0 +1,51 @@
# author: Justin Guese, 11.3.22, justin@datafortress.cloud
from fastapi import HTTPException
from fastapi import APIRouter
from pydantic import BaseModel
from typing import Dict, List
from os import environ
import glob
# reads in secrets to environment variables, such that they can be
# easily used with environ["SECRET_NAME"]
def readSecretToEnv(secretpath):
secretname = secretpath.split('/')[-1]
with open(secretpath, "r") as f:
environ[secretname] = f.read()
for secret in glob.glob("/var/openfaas/secrets/*"):
readSecretToEnv(secret)
router = APIRouter()
# just as an example
class User(BaseModel):
id: int
name: str
age: int
colleagues: List[str]
class ResponseModel(BaseModel):
data: Dict
# user: User
# otherStuff: str
@router.post("/", response_model = ResponseModel, tags=["Main Routes"])
def handle(request: Dict):
"""handle a request to the function
Args:
req (dict): request body
"""
try:
res = ResponseModel(data=request)
except Exception as e:
raise HTTPException(status_code=500, detail=str(repr(e)))
return res
@router.get("/", response_model = ResponseModel, tags=["Main Routes"])
def get():
return ResponseModel(data={"message": "Hello from OpenFAAS!"})
# again just as an example, delete this if not required
@router.get("/users/{user_id}", response_model = User, tags=["Main Routes"])
def getUser(user_id: int):
return User(id = user_id, name="Exampleuser", age=20, colleagues=["Colleague 1", "Colleague 2"])

View File

@@ -0,0 +1,19 @@
# author: Justin Guese, 11.3.22, justin@datafortress.cloud
from os import environ
import glob
from fastapi import FastAPI, Request
from fastapi.openapi.docs import get_swagger_ui_html
from function.handler import router
app = FastAPI()
app.include_router(router)
# required to render /docs path
@app.get("/docs", include_in_schema=False)
async def custom_swagger_ui_html(req: Request):
root_path = req.scope.get("root_path", "").rstrip("/")
openapi_url = root_path + app.openapi_url
return get_swagger_ui_html(
openapi_url=openapi_url,
title="API",
)

View File

@@ -0,0 +1,2 @@
fastapi
uvicorn

View File

@@ -0,0 +1,2 @@
language: python3-fastapi
fprocess: uvicorn index:app --workers 1 --host 0.0.0.0 --port 8000

View File

@@ -0,0 +1,63 @@
ARG PYTHON_VERSION=3.11
FROM --platform=${TARGETPLATFORM:-linux/amd64} ghcr.io/openfaas/of-watchdog:0.9.10 as watchdog
FROM --platform=${TARGETPLATFORM:-linux/amd64} python:${PYTHON_VERSION}-alpine as build
COPY --from=watchdog /fwatchdog /usr/bin/fwatchdog
RUN chmod +x /usr/bin/fwatchdog
ARG ADDITIONAL_PACKAGE
# Alternatively use ADD https:// (which will not be cached by Docker builder)
RUN apk --no-cache add openssl-dev ${ADDITIONAL_PACKAGE}
# Add non root user
RUN addgroup -S app && adduser app -S -G app
RUN chown app /home/app
USER app
ENV PATH=$PATH:/home/app/.local/bin
WORKDIR /home/app/
COPY --chown=app:app index.py .
COPY --chown=app:app requirements.txt .
USER root
RUN pip install --no-cache-dir -r requirements.txt
# Build the function directory and install any user-specified components
USER app
RUN mkdir -p function
RUN touch ./function/__init__.py
WORKDIR /home/app/function/
COPY --chown=app:app function/requirements.txt .
RUN pip install --no-cache-dir --user -r requirements.txt
#install function code
USER root
COPY --chown=app:app function/ .
FROM build as test
ARG TEST_COMMAND=tox
ARG TEST_ENABLED=true
RUN [ "$TEST_ENABLED" = "false" ] && echo "skipping tests" || eval "$TEST_COMMAND"
FROM build as ship
WORKDIR /home/app/
#configure WSGI server and healthcheck
USER app
ENV fprocess="python index.py"
ENV cgi_headers="true"
ENV mode="http"
ENV upstream_url="http://127.0.0.1:5000"
HEALTHCHECK --interval=5s CMD [ -e /tmp/.lock ] || exit 1
CMD ["fwatchdog"]

View File

@@ -0,0 +1,7 @@
def handle(req):
"""handle a request to the function
Args:
req (str): request body
"""
return req

View File

@@ -5,7 +5,6 @@ from .handler import handle
# To disable testing, you can set the build_arg `TEST_ENABLED=false` on the CLI or in your stack.yml
# https://docs.openfaas.com/reference/yaml/#function-build-args-build-args
def test_handle():
# assert handle("input") == "input"
pass

View File

@@ -0,0 +1,41 @@
# Copyright (c) Alex Ellis 2017. All rights reserved.
# Licensed under the MIT license. See LICENSE file in the project root for full license information.
from flask import Flask, request
from function import handler
from waitress import serve
import os
app = Flask(__name__)
# distutils.util.strtobool() can throw an exception
def is_true(val):
return len(val) > 0 and val.lower() == "true" or val == "1"
@app.before_request
def fix_transfer_encoding():
"""
Sets the "wsgi.input_terminated" environment flag, thus enabling
Werkzeug to pass chunked requests as streams. The gunicorn server
should set this, but it's not yet been implemented.
"""
transfer_encoding = request.headers.get("Transfer-Encoding", None)
if transfer_encoding == u"chunked":
request.environ["wsgi.input_terminated"] = True
@app.route("/", defaults={"path": ""}, methods=["POST", "GET"])
@app.route("/<path:path>", methods=["POST", "GET"])
def main_route(path):
raw_body = os.getenv("RAW_BODY", "false")
as_text = True
if is_true(raw_body):
as_text = False
ret = handler.handle(request.get_data(as_text=as_text))
return ret
if __name__ == '__main__':
serve(app, host='0.0.0.0', port=5000)

View File

@@ -0,0 +1,3 @@
flask
waitress
tox==3.*

View File

@@ -0,0 +1,2 @@
language: python3-flask
fprocess: python index.py