Differentiable Programming: Theory, Function Fitting, and Practical Implementations

Researchers point out that scientific computing and machine learning both rely heavily on linear algebra, suggesting a new programming paradigm called differentiable programming. This article explores the concept, its relationship with automatic differentiation, and practical applications.

What is Differentiable Programming? It extends traditional programming by embedding automatic differentiation into the language, allowing programs to be optimized via gradient descent. The Julia paper "A Differentiable Programming System to Bridge Machine Learning and Scientific Computing" is cited as a pioneering work.

Example: Elastic Damping Animation

In front‑end development, a damping function is often used to create smooth UI animations. The original JavaScript implementation is:

function damping(x, max) { let y = Math.abs(x); y = 0.82231 * max / (1 + 4338.47 / Math.pow(y, 1.14791)); return Math.round(x < 0 ? -y : y); }

By fitting the function with data from a 4‑parameter logistic (4PL) model, a refined version is obtained:

y = 0.821 * max / (1 + 4527.779 / Math.pow(y, 1.153));

Julia Example

Using Julia’s Zygote package, a loss function for a point light source is defined and differentiated:

julia> guess = PointLight(Vec3(1.0), 20000.0, Vec3(1.0, 2.0, -7.0)) julia> function loss_function(light) rendered_color = raytrace(origin, direction, scene, light, eye_pos) rendered_img = process_image(rendered_color, screen_size.w, screen_size.h) return mean((rendered_img .- reference_img).^2) end julia> gs = gradient(x -> loss_function(x, image), guess)

Swift Proposal

Swift also introduces differentiable programming to build “intelligent applications” that adapt UI behavior based on user preferences using gradient descent.

Gradient Descent Pseudocode (Swift‑like)

struct Observation { var podcastState: PodcastState var userSpeed: Float } func meanError(for model: PodcastSpeedModel, _ observations: [Observation]) -> Float { var error: Float = 0 for observation in observations { error += abs(model.prediction(for: observation.podcastState) - observation.userSpeed) } return error / Float(observations.count) } for _ in 0..<1000 { let gradient = gradient(at: model) { meanError(for: $0, observations) } model -= 0.01 * gradient }

Polynomial Fitting

A cubic polynomial z = x·w1 + x²·w2 + x³·w3 + b is fitted to the same data using TensorFlow’s automatic differentiation. The training loop (simplified) is:

optimizer = tf.keras.optimizers.Adam(0.1) t_x = tf.constant(train_x, dtype=tf.float32) t_y = tf.constant(train_y, dtype=tf.float32) wa = tf.Variable(0., dtype=tf.float32, name='wa') wb = tf.Variable(0., dtype=tf.float32, name='wb') wc = tf.Variable(0., dtype=tf.float32, name='wc') wd = tf.Variable(0., dtype=tf.float32, name='wd') variables = [wa, wb, wc, wd] for e in range(num): with tf.GradientTape() as tape: y_pred = wa*t_x + wb*tf.pow(t_x,2) + wc*tf.pow(t_x,3) + wd loss = tf.reduce_sum(tf.square(y_pred - t_y)) grads = tape.gradient(loss, variables) optimizer.apply_gradients(zip(grads, variables))

Neural Network Approach

A dense neural network with ten hidden layers (10 units each, SELU activation) is trained on the same data:

model = tf.keras.Sequential() model.add(tf.keras.layers.Dense(10, input_dim=1, activation='selu')) # … repeat for 9 more hidden layers … model.add(tf.keras.layers.Dense(1, activation='selu')) model.compile(optimizer='adam', loss='mse') history = model.fit(t_x, t_y, epochs=2000)

The network achieves a loss of ~0.33 and predicts values very close to the ground truth, though the fitted curve is less smooth than the analytical 4PL model.

Exporting the Model to the Browser

The trained TensorFlow model is saved with model.save('saved_model/w4model') , converted to TensorFlow.js format using:

tensorflowjs_converter --input_format=tf_saved_model \ --output_node_names="w4model" \ --saved_model_tags=serve ./saved_model/w4model ./web_model

In a web page, the model is loaded and executed:

import * as tf from "@tensorflow/tfjs"; import { loadGraphModel } from "@tensorflow/tfjs-converter"; tf.setBackend("webgl"); // enable GPU acceleration const model = await loadGraphModel('model.json'); const testData = tf.tensor([[0],[500],[1000],[1500],[2500],[6000],[8000],[10000],[12000]]); const outputs = model.execute(testData);

This reduces inference time from ~257 ms to ~131 ms (and down to ~78 ms on subsequent runs) thanks to WebGL acceleration.

Discussion

The article emphasizes that while differentiable programming and machine‑learning‑based fitting provide flexible solutions, they lack the interpretability of analytically derived formulas. For problems where a clear physical model exists, traditional function fitting is preferable; for complex, high‑dimensional tasks, neural networks become valuable.