# How to "maximize" an unknown categorical function?

**URL:** <https://discourse.julialang.org/t/how-to-maximize-an-unknown-categorical-function/83641>\
**Category:** Machine Learning\
**Tags:** optimization\
**Created:** [July 1, 2022, 8:04pm UTC](https://discourse.julialang.org/t/how-to-maximize-an-unknown-categorical-function/83641 "2022-07-01T20:04:49Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![Juan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/juan/32/7657_2.png) [@Juan](https://discourse.julialang.org/u/Juan)\
**Post date:** [July 1, 2022, 8:04pm UTC](https://discourse.julialang.org/t/how-to-maximize-an-unknown-categorical-function/83641/1 "2022-07-01T20:04:49Z")

</div>

Or…  
How to numerically find the value of a set of variables that produce a given condition?  
  
  
  
Say I have a problem in which, under certain conditions, something happens.  
For example, a machine that stops working.  
Or a sensor which, for some reason, starts to produce strange values.  
Or a person who suffers a heart attack.

I want to know why this happened or it’s going to happen.

I have a table with several input variables (continuous and/or discrete, maybe including time) and one output.  
This table contains observations from many experiments (or devices or people).  
I want to find what input is more likely to produce that output (the categorical output representing the extreme condition).

I don’t have a parametric function or specific model. Then I will try to fit a decision tree or a neural network.  
Then I would use that fitted model as a blackbox function as if I had an optimization problem, which is likely to be noisy, not convex and mixing integers and continuous variables.

Is there any other easier or more direct way to solve this problem?

---

<div class="post-metadata">

**Author:** ![juliohm](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/juliohm/32/215266_2.png) [@juliohm](https://discourse.julialang.org/u/juliohm)\
**Post date:** [July 1, 2022, 8:19pm UTC](https://discourse.julialang.org/t/how-to-maximize-an-unknown-categorical-function/83641/2 "2022-07-01T20:19:04Z")

</div>

> [@Juan](#):
>
> I have a table with several input variables (continuous and/or discrete, maybe including time) and one output.  
> This table contains observations from many experiments (or devices or people).  
> I want to find what input is more likely to produce that output (the categorical output representing the extreme condition).

If you have a binary response (extreme vs. non-extreme condition), you can investigate the sensitivity of this response variable to all other variables in the table. The question then becomes: how sensitive is my response to variable X1, X2, … or to a combination of variables X1-X2, X1-X3, …

> [@Juan](#):
>
> I don’t have a parametric function or specific model. Then I will try to fit a decision tree or a neural network.

If you can afford a model like a decision tree, then the sensitivity analysis question is trivial. You can check which variables are chosen as the first features in the tree and which are chosen as the last. Alternatively, you can use a generic method such as Shapley values that works with any machine learning model:

> **[GitHub - nredell/ShapML.jl: A Julia package for interpretable machine...](https://github.com/nredell/ShapML.jl)**
>
> A Julia package for interpretable machine learning with stochastic Shapley values - GitHub - nredell/ShapML.jl: A Julia package for interpretable machine learning with stochastic Shapley values

---

<div class="post-metadata">

**Author:** ![Juan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/juan/32/7657_2.png) [@Juan](https://discourse.julialang.org/u/Juan)\
**Post date:** [July 4, 2022, 1:08am UTC](https://discourse.julialang.org/t/how-to-maximize-an-unknown-categorical-function/83641/3 "2022-07-04T01:08:36Z")

</div>

OK thank you, I will investigate about interpretable machine learning.  
Anyway that seems useful to select the variables that have a larger effect on the output but not for finding the best values.

And what if the output doesn’t depend just on the current values but on the pattern of values that has already happened?  
I guess then I would also need to use time lagged versions of the variables or use some kind of memory. Maybe LSTM networks, I haven’t used them before.

---

<div class="post-metadata">

**Author:** ![jling](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jling/32/212909_2.png) [@jling](https://discourse.julialang.org/u/jling)\
**Post date:** [July 4, 2022, 2:14am UTC](https://discourse.julialang.org/t/how-to-maximize-an-unknown-categorical-function/83641/4 "2022-07-04T02:14:14Z")

</div>

maybe [GitHub - robertfeldt/BlackBoxOptim.jl: Black-box optimization for Julia](https://github.com/robertfeldt/BlackBoxOptim.jl/)

---

<div class="post-metadata">

**Author:** ![maxkapur](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/maxkapur/32/21208_2.png) [@maxkapur](https://discourse.julialang.org/u/maxkapur)\
**Post date:** [July 4, 2022, 6:22am UTC](https://discourse.julialang.org/t/how-to-maximize-an-unknown-categorical-function/83641/5 "2022-07-04T06:22:36Z")

</div>

No need to reach so quickly for a computationally expensive technique like mixed-integer programming or a neural network!

If your goal is to predict “whether an event happens” (Y = 0 or 1) based on “state information” about the model (various numbers X\_j), and you have observations data in the form of y\_i and x\_{ij} for j = 1 \dots m, then you can just use logistic regression.

> **[Logistic regression](https://en.wikipedia.org/wiki/Logistic_regression)**
>
> In statistics, the (binary) logistic model (or logit model) is a statistical model that models the probability of one event (out of two alternatives) taking place by having the log-odds (the logarithm of the odds) for the event be a linear combination of one or more independent variables ("predictors"). In regression analysis, logistic regression (or logit regression) is estimating the parameters of a logistic model (the coefficients in the linear combination). Formally, in binary logistic reg Bi...

This fits a function of the form

P(Y = 1) = \frac{1}{1+ \exp( \beta \cdot X)}

with parameter \beta, and (assuming the input data X\_j are not terribly intercorrelated) the value of \beta\_j is easy to interpret: The sign tells you if it increases or decreases the probability of Y = 1, and the magnitude tells you how much. Estimating \beta by maximum likelihood requires only solving a concave maximization problem.

The fact that some of your X\_j are categorical can be handled using [dummy coding](https://stats.oarc.ucla.edu/other/mult-pkg/faq/general/faqwhat-is-dummy-coding/).

---

<div class="post-metadata">

**Author:** ![Juan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/juan/32/7657_2.png) [@Juan](https://discourse.julialang.org/u/Juan)\
**Post date:** [July 5, 2022, 11:44am UTC](https://discourse.julialang.org/t/how-to-maximize-an-unknown-categorical-function/83641/6 "2022-07-05T11:44:50Z")

</div>

But a logistic model can work well if the most probable event is located towards the lower values of X or towards the higher values (monotonically decreasing or increasing).  
If our data show a maximum around intermediate X values, or multiple maximums, or something more complex (such as dependency on past values), then it won’t work.

---

<div class="post-metadata">

**Author:** ![maxkapur](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/maxkapur/32/21208_2.png) [@maxkapur](https://discourse.julialang.org/u/maxkapur)\
**Post date:** [July 5, 2022, 12:00pm UTC](https://discourse.julialang.org/t/how-to-maximize-an-unknown-categorical-function/83641/7 "2022-07-05T12:00:02Z")

</div>

Sure, but you can build these kinds of features into the model by including quadratic or higher-order terms (which has no effect on the difficulty of estimation). Logistic regression isn’t a magic swiss army knife that will solve everything, but it’s better to start with a simple model and add complexity gradually as you need it than the opposite, right?

---

<div class="post-metadata">

**Author:** ![Juan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/juan/32/7657_2.png) [@Juan](https://discourse.julialang.org/u/Juan)\
**Post date:** [July 5, 2022, 5:24pm UTC](https://discourse.julialang.org/t/how-to-maximize-an-unknown-categorical-function/83641/8 "2022-07-05T17:24:50Z")

</div>

I agree.

Anyway fitting the data with a logistic model will create a model to predict new Y values from X, but won’t tell me what X values are more likely to produce that Y=1.

How should I do it?  
Just calculating the derivative of the polynomial fitted function and equating to zero?  
Or I need to use something more complex?

---

<div class="post-metadata">

**Author:** ![ElOceanografo](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/eloceanografo/32/624_2.png) [@ElOceanografo](https://discourse.julialang.org/u/ElOceanografo)\
**Post date:** [July 5, 2022, 9:11pm UTC](https://discourse.julialang.org/t/how-to-maximize-an-unknown-categorical-function/83641/9 "2022-07-05T21:11:05Z")

</div>

> Just calculating the derivative of the polynomial fitted function and equating to zero?

Exactly. The model for a quadratic logistic regression with a 0/1 response variable Y and a continuous predictor X is

```julia
Y ~ Bernoulli(p)
p = logistic(u)
u = a + b*x + c*x^2

```

so once you’ve got fitted values for a, b, and c you can just solve for the maximum.