39. The Process of Creating a Computational System: A Collective Audiovisual Composition

Background

function sonoficarPlastica() {

  loadPixels(); // Loads all pixels in an array. Function of p5

  var filasConNegro = []

  var frequencias = []

  //tiempoApixel converts the pixels into a time relationship

  var timeActualPix = int(min(tiempoApixel() + 3, width));

  // The following code goes through all pixels column by column

  // in each column it scrolls through all the rows and if a

  // pixel in black adds it to a list where the

  // coordinates with black pixels.

  for (var cont = 0; cont < height; cont++) {

    var pixCanalRojo = pixels[(cont * width + timeActualPix) * 4 + 0];

    if (pixCanalRojo == 0) {

      filasConNegro.push(height - cont);

    }

  }

  //The frequency corresponding to each coordinate is obtained.

  //that has black in frequency format.

  frequencias = filasConNegro.map(pixNegro => {

    let midi = map(pixNegro, 0, height, notaGrave, notaAguda);

    //midi2freq converts the midi note to its corresponding frequency

    return midi2freq(midi);

  });

  // For each frequency a 20 millisecond oscillator is generated.

  for (var i = 0; i < frequencias.length; i++) {

    var osc = new p5.Oscillator(frequencias[i], ‘sine’);

    // amp, start and stop are functions of the webaudioapi.

    osc.amp(0);

    osc.start();

    osc.amp(0.1, 0.05);

    osc.amp(0, 0.1, 0.05);

    osc.stop(0.2);

  }

}

Code 39.1

The second type of “sonification,” called ”parametric,” reads and stores the trace information in time and frequency parameters, then assigns it to a single oscillator per line that modulates its values over time using envelopes in its frequency and amplitude parameters. When the “touch” button is pressed, the oscillators are activated and change their frequency according to the movement of the base generator trace (see Code 39.2).

function sonificarParametrica() {

  destruyeOsc(); // Release all active generators with stop

  //del webaudioapi.

  osciladores = [];

  // compositionTF is an array of arrays in which

  //each internal array contains the stroke points.

  for (var i = 0; i < composicionTF.length; i++) {

    var gestoTMP = composicionTF[i];

    var osc = new p5.Oscillator(composicionTF[i][1][0], ‘sine’);

    osc.start(composicionTF[i][0][0]);

    // A single oscillator is generated whose frequency changes

    // discretely but at a speed that gives us

    osc.amp(0.0, 0, composicionTF[i][0][0]);

    osc.amp(0.1, 0.1, composicionTF[i][0][0]);

    // nesting for that goes through each point of each stroke and assigns

    //frequency at the corresponding moment (time).

    for (var j = 1; j < gestoTMP[0].length; j++) {

      var tTmp = composicionTF[i][0][j];

      var fTmp = composicionTF[i][1][j];

      var tTmpAnterior = composicionTF[i][0][j - 1];

      osc.freq(fTmp, tTmp - tTmpAnterior, tTmpAnterior);

    }

    // On the last point of each stroke the volume is turned down and off.

    var indxtUltimo = gestoTMP[0].length - 1;

    osc.amp(0, 0.1, gestoTMP[0][indxtUltimo] - 0.1);

    osc.stop(gestoTMP[0][indxtUltimo]);

    // All oscillators are added to one array

    //to be able to turn them off in another external function.

    osciladores.push(osc);

  }

}

Code 39.2

Another feature of the project is the incorporation of a grid in both the X and Y axes that can be activated or deactivated, depending on the accuracy with which you want to manage the times and heights (pitches) in the traces.

Derived from this first exploration, a second project was generated throughout 2022, which we named Sound Flock. We will now jointly present these projects below.

The Collective Imaginary as a Triggering Process

Within the 3D space, the principle that the absolute interpretation of the positions of the agents is accompanied by a reading from any viewing angle, so that the user may rotate and move within the 3D environment at will. It is important to note that the user may also activate planes from any point in space, moving towards the background.

Development of the Work

The development of the work was guided by the vision raised in the investigation with the aforementioned factors: agentiality, flock behavior, and interference of planes on the space. The general purpose was to have an audiovisual experience in which the events, and their ranges and parameters, were manipulated by means of a control panel. In this way, a resultant was planned that could produce “compositions” to be downloaded, with the objective of preserving them in fixed format for their future and possible diffusion.

In order to develop and achieve the proposed objectives, it was essential to work step by step with the following elements:

1. Define the 3D space.

The first decision was to establish a 3D space, which we called “external space,” which corresponds to the Cartesian absolute coordinates of the digital graphic system. Within this virtual space, a cube was placed whose function is to be the container for the graphical representation of the agents.

2. Characteristics of the agents.

During this stage, work was done on the graphic representation of the agents and the following was determined:

3. Agent behavior.

2. Sound capture by intersecting planes.

At this stage, a mechanism of reading by means of blueprints was developed in relation to the position of the agents. This reading is generated when the plane crosses the cube, reading the relative position of each of the agents, adding a new sound. This procedure was performed by identifying the relative coordinates of each agent in space.

3. Control panel.

Within the original conception, one of the goals was to allow the user to interact with the system and thus achieve a more interactive experience. With this objective in mind, a control panel was implemented that allows the manipulation of a large number of parameters and ranges that help to control the sound results.

Emerging Behaviors

Flocking Behavior

The separation function consists of how agents avoid colliding with each other, avoiding and pushing each other according to a given range. The function has two important local variables: “desiredseparation,” which consists of the level of separation of the agent from the rest (in other words, this impulse indicates what the range of action of the function is), and “steer,” which indicates a vector with the future direction and position of the agent. To know which of the agents are in the range of action of the function, the distance between each agent and the rest must be calculated. If the distance is greater than the “desiredseparation” then three operations are applied: the vector of the agent must be subtracted from the vector of the nearby agent, the subtraction normalized, and then divided scalarly with the value of the distance. Finally, this value is added to the variable “steer.” This process is repeated with each of the agents, as long as their distance from the agent is less than the “desiredseparation.”

The second part of the function consists of dividing the overall “steer” by the number of times that the previous operation was performed; i.e., normalize the value of “steer,” and finally adapt this value to the general properties of the agents (in other words, adapt to the maximum speed, the current speed, and the maximum force). Thus, we finally have the “steer” to be applied to the agent in question.

The alignment function instructs each agent to change its direction of movement according to the position of the rest of the agents, causing groups of agents to move in the same direction. Like the separation function, the alignment function has two important variables: “neighbordist” and “sum.” “Neighbordist,” is the distance at which the function will come into effect, and “sum” is a vector that will indicate the direction and final position of the agent. For each agent within the range of “neighbordist,” the current velocity of each of them will be added to “sum.”

The second part of the function consists of taking the value of “sum” (the sum of all velocities) and dividing it by how many agents are within the range of action of the function; i.e., normalize such division and adapt it to the general properties of the agent and, finally, return the value of “sum.” In case there is no agent within the range of action, a vector with values of 0 in each axis must be returned. In this way, the behavior of the agent will not be affected in any way.

The cohesion function acts on the agents to calculate the average position of nearby agents and instructs them to move to the average position. Like the separation and alignment functions, the cohesion function has two parts: in the first part, a variable (again defined as “sum”) is obtained that sums all the positions within the action range (as in alignment, the action range is defined by the variable “neighbordist”). In the second part, this sum is divided into the number of agents within the “neighbordist.” Before returning the value of “sum,” the secondary function “seek” must be applied, which adapts the value to obtain the final result.

Finally, these three functions are added together with the “applyForce” function, to indicate to each agent its next position in each interaction of the animation. In this way, one has a new position that summarizes how separated and aligned the agent will be with respect to the others and if its direction has cohesion with the rest.

According to different configurations of these functions, movements will be obtained that are either more synchronous or more chaotic. It should be noted that some functions can override the effect of others; for example, in the case of alignment with separation, a high level of separation and cohesion will generate a hesitant movement of the agents.

Synchronization

Two different strategies were used to achieve synchronization: pulse synchronization and phase synchronization. For this purpose, two different functions were used: “bpmsync(),” which is in charge of synchronizing the pulses, and “syncronization(),” which is in charge of synchronizing the phases (see code 39.3).

    let sigma = 0;

    for (let i = 0; i < boids.length; i++) {

      sigma += Math.sin(boids[i].paso - this.paso);

    }

    let term2 = 1 / boids.length * sigma;

    this.paso += term2 * 0.001; // Value that reduces the speed at which synchronization is done.

  }

  syncronization(boids) {

    if (this.estado == “DESPIERTO”) {

      for (let i = 0; i < boids.length; i++) {

        let d = this.position.distanceTo(boids[i].position);

        // If the distance is greater than 0 and less than an arbitrary amount (0 when you are yourself).

        if ((d > 0) && (d < SYNC_DISTANCIA)) {

          if (boids[i].estado == “DESPIERTO”) {

            this.fase += this.paso;

          }

        }

      }

    }

  }

Code 39.3

Individual Comments

As for my artistic work, the creation of sound by computer has implied a significant enrichment since it has forced me to consider paradigm changes. For example, I now treat the computer as just another instrument or as an extension of traditional instruments. This allows me to materialize in a more reliable way what I need. On the other hand, programming and its field of action invites me to consider areas of knowledge that are not necessarily limited only to sound, so it can be a useful tool to create multidisciplinary works in which the artist’s perceptive capacities are completed, without restricting them to a single sense.

The concept of work/tool within a digital platform is appealing to me. I had never worked on such a project before. One continually thinks about how the audience—or perhaps, in this case, the solitary virtual surfer—perceives the work; but one also thinks about how the composer operates the tool to get something to their liking. Experimenting with “the form of the work equals the utility of the tool” or “the final version of the work equals the audience’s experimentation with the tool” were attractive approaches during the process. One never knows how far the audience will take the proposal. Will it achieve what was imagined? Can one say that there can be a misuse of the tool? Delegating decisions to someone else is always a bit of a mystery.

In this way, the result described in these pages was achieved: a digital system for audiovisual sound composition, created collectively. Undoubtedly, this result can grow and be refined in its details, but, at the same time, I celebrate that a functional system was achieved, derived from an intention of collective creation that served as a learning process. With this, I can say that we obtained a real, tangible audiovisual system, as well as a sound reality.

Sound Examples Available

Conclusions

In the territory of sound, we manage to control and listen to processes that are at a particular boundary, as they are neither deterministic, on the one hand, nor chaotic, on the other. The tool is at a point where control of the resulting sonorities is maintained, although several of the processes are not directly controlled by the user. At this stage the user can suggest ranges, dynamics, and interaction situations, but the particles have the same control over the resulting sonorities and a certain “creative independence.”

What is presented here is the materialization of one possibility among many others. It is the sum of thoughts, needs, and curiosities of the teacher and the students who, in a collective search for a digital composition tool, arrived at a specific synthesis which allows a particular—and hopefully original—exploration for the creation of sound structures with the particularities described in this chapter. Therefore, this chapter is a description of a methodological work guided by the fantasies and curiosities of all the members of the group in an analogous way to the agents of the system.

Future Work

As part of the development and improvement of this project, several points are planned to achieve a greater sonic and perceptual richness and, thus, to grow and improve the users’ experiences. Among the points to be implemented in the future, the following stand out:

1. Change the reading of the agent values on the axes X,Y, Z from the center to the ends to achieve a better sound balance.
2. Incorporate the binaural and/or multichannel audibility of the system, and consequently enable the download of a recording with such characteristics.
3. Clean and refine the reading of the plans on the 3D space.
4. To be able to change the waveform of the agents, as well as to incorporate the option of uploading an audio file to be used as a sample for sound generation.
5. Consider the possibility of incorporating not only a cube as a delimiter of the agents’ space, but also other geometric figures.
6. To have the possibility of having delimiters within other delimiters, such as, for example, cubes inside cubes.

References