# Questions tagged [ml]

773 questions

1

votes

0

answer

11

Views

### How exactly tf.estimator.DNNClassifier calculate Output/Logits?

i am new in Machine Learning here,
just try to experimenting with tensorflow to understand some fundamental concept.
I want to ask about how exactly tensorflow calculate output/logits layer, especially with Pre-Made models like tf.estimator.DNNClassifier.
When i try to see the logits value from tens...

1

votes

1

answer

2.2k

Views

### Customize Distance Formular of K-means in Apache Spark Python

Now I'm using K-means for clustering and following this tutorial and API.
But I want to use custom formula for calculate distances. So how can I pass custom distance functions in k-means with PySpark?

1

votes

1

answer

558

Views

### Spark (Java) transformSchema() in custom Transformer

I want to use my custom transformer along with StandardScaler:
VectorizerTransformer vectorizerTransformer = new VectorizerTransformer(field.getName());
pipelineStages.add(vectorizerTransformer);
StandardScaler scaler = new StandardScaler()
.setInputCol(vectorizerTransformer.getOutputColumn())
.setO...

1

votes

2

answer

324

Views

### Error in Tensorflow Object Detection running on Cloud ML: no module object_detection.train

Not sure if this is a bug (file report), or I did something wrong.
System information:
Linux 17.04
TensorFlow version:
1.9.0
Python version:
2.7.13
Command I used:
gcloud ml-engine jobs submit training object_detection_$(date +%Y%m%d_%H%M%S) \
--job-dir="gs://mybucket/train" \
--packages dist/obje...

1

votes

1

answer

39

Views

### How to tune hyperparameters using custom model evaluation algorithm?

In my model evaluation algorithm I would like to get model predictions for validation data and apply an algorithm which models and imitates some real-world scenarios based on validation data and predictions.
In my scenario evaluation algorithm depends not only on true target values (y_true) and pred...

1

votes

2

answer

67

Views

### Access from gcloud ml-engine jobs to Big Query

I have a python ML process which connects to BigQuery using a local json file which the env variable GOOGLE_APPLICATION_CREDENTIALS is pointing to (The file contains my keys supplied by google, see authentication getting-started )
When Running it locally its works great.
Im now looking to deploy my...

1

votes

1

answer

28

Views

### How to deploy templates to different schema databases?

I have staging and final databases, both currently sharing a schemas database. We're using TDE, where some templates apply to staging and some to final.
Can ml-gradle deploy some templates to one schemas database and others to a second?

1

votes

1

answer

186

Views

### Defining a normal form with Haskell

I'm trying to get an expression and turn it into a standard form. To better clarify my purpose, assume you define a general style for your expressions like this:
∑(a*b) // sum of products
Now if you're given an input which is not in that format like: (a+b)*(c+d), you'll need to normalize it firs...

1

votes

1

answer

144

Views

### (Beginner's) issue with redundant case statement in SML

I'm trying to write a function in SML to compute the partial sum of an alternating harmonic series, and for the life of me I can't figure out why the compiler says one of the cases is redundant. I haven't used case statements before(or local, for that matter), but the order of these cases seems rig...

1

votes

1

answer

3.2k

Views

### ERROR: Inserting EQUALOP

Hi can someone help me figure why i get the
ERROR: Inserting EQUALOP
for the following code?
fun generator inchan outchan n = if n>0
then
(let
fun loop () =
val c = recv(outchan)
val _ = send (inchan, c)
in
(loop ();(generator inchan outchan (n...

1

votes

1

answer

212

Views

### how to typecast a value in ocaml

I am trying to typecast a value from one type another.
type days =
| Mon | Tue | Wed | Thu | Fri | Sat
type ('a, 'b) holidays = H of 'a | W of 'b
(* H: holiday, W: Working *)
let my_list = [H Mon; H Tue; W Wed]
From the above my_list, I would like to extract, "Mon" and not "H Mon". How can I go abou...

1

votes

1

answer

89

Views

### Can you do typecasting in ML?

I haven't done any programming in ML, but I am studying functional programming. ML is really particular about types in general, but I wanted to know if you could do typecasting to get around its strict type limitations? If you can't, is it because ML wants to lessen the chances of incorrect types, o...

-1

votes

1

answer

89

Views

### Update Machine Learning models in Mllib dataframe based PySpark (2.2.0)

I have built a Machine Learning model based on clustering, & now just want to update it with new data periodically (on daily basis). I am using PySpark Mlib, & not able to find any method in Spark for this need.
Note, required method 'partial_fit' available in scikit-learn, but not in Spark.
I am no...

0

votes

0

answer

8

Views

### Keras Load_module : Unknown Layer:Name

The following is my model that I am using when training
model = Sequential()
model.add(Conv2D(64, kernel_size=3, use_bias=False,
input_shape=(size, size, 1)))
model.add(BatchNormalization())
model.add(Activation("relu"))
model.add(Conv2D(32, kernel_size=3, use_bias=False))
model.add(BatchNormalizat...

1

votes

0

answer

10

Views

### UWP: The type or namespace name 'TensorFloat' could not be found

I am trying to run a UWP app I made (which worked) on a different computer, and I am now getting the error:
The type or namespace name 'TensorFloat' could not be found (are you
missing a using directive or an assembly reference?)
I have VS 2017, my Windows 10 version is 1809, build 17763.194, and I...

0

votes

0

answer

2

Views

### ML.NET - Multiclass Classificaion score values

I currently have a project to take large bits of text and classify them as types. This is similar to the sentiment sample provided by microsoft except its multiclass instead of binary.
I have the code working just fine and will likely become stronger as we add data to it. However, i have hit a snag...

2

votes

1

answer

66

Views

### Getting worse results on gcloud vs local training

I'm trying to understand why my local results are better than the gcloud results.
In local, I run a job like this:
gcloud ml-engine local train --module-name trainer.task --package-path trainer -- --vocabulary-file trainer/data/vocab.txt --class-files $CLASS_FILES --job-dir trainer/lr0001 --num-epoc...

2

votes

1

answer

40

Views

### int -> ('a -> 'a)

I have a question that I came across studying SML. It requires the return type to be int -> (’a -> ’a). This is what I have done:
- fn x:int => fn y => y;
val it = fn : int -> 'a -> 'a
How can I make the return type to be int -> ('a -> 'a) where 'a -> 'a is in brackets? Any help would be appreci...

2

votes

2

answer

97

Views

### StandardML 97 definition

Actual Standard ML specification is The Definition of Standard ML (Revised)
, which available on MITPress only in a print version.
Is it availaible in electronic format (pdf, ps, i.e.)?

9

votes

1

answer

494

Views

### Spark ML - MulticlassClassificationEvaluator - can we get precision/recall by each class label?

I am doing a multiclass prediction with random forest in Spark ML.
For this MulticlassClassificationEvaluator() in spark ML, is it possible to get precision/recall by each class labels?
Currently, I am only seeing precision/recall combined for all class together.

3

votes

2

answer

1.3k

Views

### Does Spark.ml LogisticRegression assumes numerical features only?

I was looking at the Spark 1.5 dataframe/row api and the implementation for the logistic regression. As I understand, the train method therein first converts the dataframe to RDD[LabeledPoint] as,
override protected def train(dataset: DataFrame): LogisticRegressionModel = {
// Extract columns from d...

3

votes

3

answer

1.2k

Views

### How to Split the Predicted Probabilities Produced by ML Pileline Logistic Regression

I'm trying to extract the predicted probability from the logistic model using ML pipeline and DataFrame API. The output of predicted probabilities is a column vector that stores the predicted probabilities for each class(0, 1) in as shown below. I wonder how I can extract only the probability for cl...

2

votes

1

answer

531

Views

### RPAREN EQALOP and unbound variable and constructor errors in SML/ML

Said I have got 2 CNF logical phrases a,b and my distrib function should return the CNF form of a|b (a OR b).
Replacing rules that I've got are:
1) Replace p|(q&r) by (p|q)&(p|r)
2) Replace (q&r)|p by (q|p)&(r|p)
A prop defined this way:
datatype prop = Atom of string | Not of prop | And of prop*p...

2

votes

1

answer

258

Views

### Returning a recursive function in ML

How can I return a recursive function in ML?
As far as I know recursive anonymous functions cannot be returned and only anonymous functions
can be used as returning value(if the returning value is function...).

1

votes

2

answer

4.2k

Views

### Logistic regression with spark ml (data frames)

I wrote the following code for logistic regression, I want to use the pipeline API provided by spark.ml. However it gave me an error after I try to print coefficients and intercepts. Also I am having trouble computing the confusion matrix and other metrics like precision, recall.
#Logistic Regress...

1

votes

1

answer

123

Views

### Type clash in Standard ML datatype with same constructor name

I need to describe in Standard-ML a language made of properties and values. My property system is made of properties which can have values, like for example:
color: red | yellow | blue | transparent
align: left | center | right
bgcolor: red | yellow | blue | transparent
I created this sml file which...

1

votes

3

answer

246

Views

### ML list typing when converting char list to int list

I've been having an issue with converting a character list to an int list. My goal is to basically take a number such as 325 and have it return a list of [3,2,5]. What I've done so far is take the number and then convert it to a string, which then gets exploded to a char array. I then want to conver...

1

votes

2

answer

115

Views

### Why automatic currying in ML family languages

ML family languages have automatic currying - where a function call with two arguments is considered to apply them one at a time, f x y = (f x) y - whereas other language families don't.
Is this just a historical accident, or is there a technical reason for this?
Note that I am not asking whether au...

16

votes

6

answer

637

Views

### Can compilers deduce/prove mathematically?

I'm starting to learn functional programming language like Haskell, ML and most of the exercises will show off things like:
foldr (+) 0 [ 1 ..10]
which is equivalent to
sum = 0
for( i in [1..10] )
sum += i
So that leads me to think why can't compiler know that this is Arithmetic Progression and use...

17

votes

3

answer

7.3k

Views

### What is the difference between HashingTF and CountVectorizer in Spark?

Trying to do doc classification in Spark. I am not sure what the hashing does in HashingTF; does it sacrifice any accuracy? I doubt it, but I don't know. The spark doc says it uses the "hashing trick"... just another example of really bad/confusing naming used by engineers (I'm guilty as well). Coun...

17

votes

4

answer

6.9k

Views

### Dropping a nested column from Spark DataFrame

I have a DataFrame with the schema
root
|-- label: string (nullable = true)
|-- features: struct (nullable = true)
| |-- feat1: string (nullable = true)
| |-- feat2: string (nullable = true)
| |-- feat3: string (nullable = true)
While, I am able to filter the data frame using
val data = ra...

17

votes

2

answer

12.6k

Views

### How to cross validate RandomForest model?

I want to evaluate a random forest being trained on some data. Is there any utility in Apache Spark to do the same or do I have to perform cross validation manually?

1

votes

2

answer

698

Views

### SML fibonacci large numbers

I tried to write my own fib function that works for large numbers (over 50) and I had no luck. First I tried the obvious solution but that overflows way to quicly. my next solution was this
$fun fib(a:int, b:int, index:int) = if(index = 1) then
$ (a+b)
$...

17

votes

4

answer

6.5k

Views

### Google Dataflow vs Apache Spark

I am surveying Google Dataflow and Apache Spark to decide which one is more suitable solution for our bigdata analysis business needs.
I found there are Spark SQL and MLlib in the spark platform to do structured data query and machine learning.
I wonder is there any corresponding solution in the Go...

1

votes

2

answer

276

Views

### Evaluator in OCaml/ML

type bool_exp =
TT
| FF
| Var of string
| And of bool_exp * bool_exp
| Not of bool_exp ;;
eval : bool_exp -> (string -> bool) -> bool
I am trying to write an evaluator function called eval. I am very new to OCaml and not used to the syntax. Where can I start in writing this?

2

votes

1

answer

294

Views

### PySpark read in a textfile as Dense vectors

I'm working with PySpark and trying to load a file of the format below, where each line is a vector of counts
[1394, 56692, 0, 10, 22]
[0, 0, 0, 0, 0]
[2235, 123, 678, 0, 999]
And I'm trying to load this into Spark using sc.textFile(path/to/counts.txt). How can I convert each row of the file to be a...

1

votes

1

answer

145

Views

### Type clash when creating a list from a set abstype in SML

I feel like this should be easier than I'm making it, but I'm having some trouble with this function that should change a set to a list.
fun explode(st(x, xs)) =
let
val listTemp = []
in
if isEmpty(xs) then listTemp::x else explode(xs)::x
end
| explode(nilset) = [];
I'm getting a type clash, saying...

1

votes

4

answer

723

Views

### how can i find partitions of given list? [closed]

how can i find all partitions of list of integers? mostly i need algorithm that uses recursion as i'm gonna implement it in SML. i need just algorithm, i will do coding part by myself. by mistake i wrote code for finding subsets and i don't have too much time left for this
SML is kinda similar pasca...

1

votes

1

answer

1.2k

Views

### How to get a string from TextIO in sml/ml?

I'm trying to read text from a file in SML. Eventually, I want a list of individual words; however, I'm struggling at how to convert between a TextIO.elem to a string. For example, if I write the following code it returns a TextIO.elem but I don't know how to convert it to a string so that I can c...

1

votes

1

answer

2.2k

Views

### SML Sum of List using option and Pattern matching

I'm creating sum of list and using option in it. When I pass an empty list, I should get NONE or else SOME value.
I'm able to do that in the following way:
fun sum_list xs =
case xs of
[] => NONE
| x =>
let
fun slist x =
case x of
[] => 0
| x::xs' => x + slist xs'
in
SOME (slist x)
end
But I want...