Using expo file system to download and cache model, using file uri crashes app

[nadeem-portico]

The app crashes when using modelPath after downloading the file from network.

const llmInference = useLlmInference({
   storageType: 'file',
   modelPath: '/data/user/0/com.offlinellmpoc/files/gemma-2b-it-cpu-int4.bin',
 });

or

const llmInference = useLlmInference({
  storageType: 'file',
  modelPath: 'file:///data/user/0/com.offlinellmpoc/files/gemma-2b-it-cpu-int4.bin',
});

[alam65]

I'm also facing the same crash issue in react native cli. @cdiddy77 can you help us with this issue

[nadeem-portico]

I was able to run it using a different model. Seems to be an issue with model compatibility with the phone used. But the crash shouldn't occur. Error out maybe.

[alam65]

I was able to run it using a different model. Seems to be an issue with model compatibility with the phone used. But the crash shouldn't occur. Error out maybe.

@nadeem-portico which model did you use?

[nadeem-portico]

I was able to run it using a different model. Seems to be an issue with model compatibility with the phone used. But the crash shouldn't occur. Error out maybe.

@nadeem-portico which model did you use?

gemma-2b-it-cpu-int8.bin

[alam65]

I was able to run it using a different model. Seems to be an issue with model compatibility with the phone used. But the crash shouldn't occur. Error out maybe.

@nadeem-portico which model did you use?

gemma-2b-it-cpu-int8.bin

Hey @nadeem-portico, I'm still getting the error
[Error: internal: Failed to initialize engine: %sINTERNAL: ; RET_CHECK failure (external/odml/odml/infra/genai/inference/utils/llm_utils/scoped_file_posix.cc:27) (fd)>=(0)open() failed: /storage/emulated/0/Download/gemma-2b-it-cpu-int8.bin]

import React from 'react';
import {
  ActivityIndicator,
  Keyboard,
  KeyboardAvoidingView,
  Platform,
  Pressable,
  SafeAreaView,
  ScrollView,
  Text,
  TextInput,
  TouchableWithoutFeedback,
  View,
} from 'react-native';
import {colors, styles} from './src/styles';
import {type Message} from './src/types';
import {useLlmInference} from 'react-native-llm-mediapipe';

const samplePrompts = [
  "Explain the difference between 'affect' and 'effect' and use both words correctly in a complex sentence.",
  'If all Roses are flowers and some flowers fade quickly, can it be concluded that some roses fade quickly? Explain your answer.',
  'A shop sells apples for $2 each and bananas for $1 each. If I buy 3 apples and 2 bananas, how much change will I get from a $10 bill?',
  "Describe the process of photosynthesis and explain why it's crucial for life on Earth.",
  'Who was the president of the United States during World War I, and what were the major contributions of his administration during that period?',
  'Discuss the significance of Diwali in Indian culture and how it is celebrated across different regions of India.',
  'Should self-driving cars be programmed to prioritize the lives of pedestrians over the occupants of the car in the event of an unavoidable accident? Discuss the ethical considerations.',
  'Imagine a world where water is more valuable than gold. Describe a day in the life of a trader dealing in water.',
  'Given that you learned about a new scientific discovery that overturns the previously understood mechanism of muscle growth, explain how this might impact current fitness training regimens.',
  'What are the potential benefits and risks of using AI in recruiting and hiring processes, and how can companies mitigate the risks?',
];

let samplePromptIndex = 0;

function App(): React.JSX.Element {
  const textInputRef = React.useRef<TextInput>(null);
  const [prompt, setPrompt] = React.useState('');
  const messagesScrollViewRef = React.useRef<ScrollView>(null);
  const [messages, setMessages] = React.useState<Message[]>([]);
  const [partialResponse, setPartialResponse] = React.useState<Message>();

  const llmInference = useLlmInference({
    storageType: 'file',
    modelPath: '/storage/emulated/0/Download/gemma-2b-it-cpu-int8.bin',
    // 'gemma-1.1-2b-it-gpu-int8.bin' or the name of the model that
    // you placed at android/app/src/main/assets/{MODEL_FILE}
  });

  const onSendPrompt = React.useCallback(async () => {
    if (prompt.length === 0) {
      return;
    }
    setMessages(prev => [...prev, {role: 'user', content: prompt}]);
    setPartialResponse({role: 'assistant', content: ''});
    setPrompt('');
    const response = await llmInference.generateResponse(
      prompt,
      partial => {
        setPartialResponse(prev => ({
          role: 'assistant',
          content: (prev?.content ?? '') + partial,
        }));
      },
      error => {
        console.error(error);
        setMessages(prev => [...prev, {role: 'error', content: `${error}`}]);
        setPartialResponse(undefined);
      },
    );
    setPartialResponse(undefined);
    setMessages(prev => [...prev, {role: 'assistant', content: response}]);
  }, [llmInference, prompt]);

  const onSamplePrompt = React.useCallback(() => {
    setPrompt(samplePrompts[samplePromptIndex++ % samplePrompts.length] ?? '');
    textInputRef.current?.focus();
  }, []);

  return (
    <SafeAreaView style={styles.root}>
      <KeyboardAvoidingView
        keyboardVerticalOffset={0}
        behavior={Platform.OS === 'ios' ? 'padding' : 'height'}
        style={styles.keyboardRoot}>
        <TouchableWithoutFeedback
          onPress={() => Keyboard.dismiss()}
          style={styles.promptInnerContainer}>
          <ScrollView
            ref={messagesScrollViewRef}
            style={styles.messagesScrollView}
            contentContainerStyle={styles.messagesContainer}
            // onContentSizeChange={() =>
            //   messagesScrollViewRef.current?.scrollToEnd()
            // }
          >
            {messages.map((m, index) => (
              <MessageView message={m} key={index} />
            ))}
            {partialResponse && <MessageView message={partialResponse} />}
          </ScrollView>
        </TouchableWithoutFeedback>
        <View style={styles.promptRow}>
          <Pressable
            onPress={onSamplePrompt}
            // disabled={prompt.length === 0 || partialResponse !== undefined}
            style={styles.samplePromptButton}>
            <Text style={styles.samplePromptButtonText}>⚡️</Text>
          </Pressable>
          <TextInput
            ref={textInputRef}
            selectTextOnFocus={true}
            onChangeText={setPrompt}
            value={prompt}
            placeholder={'prompt...'}
            placeholderTextColor={colors.light}
            multiline={true}
            style={styles.promptInput}
          />
          <Pressable
            onPress={onSendPrompt}
            // disabled={prompt.length === 0 || partialResponse !== undefined}
            style={styles.sendButton}>
            {partialResponse !== undefined ? (
              <ActivityIndicator />
            ) : (
              <Text style={styles.sendButtonText}>Send</Text>
            )}
          </Pressable>
        </View>
      </KeyboardAvoidingView>
    </SafeAreaView>
  );
}

const MessageView: React.FC<{message: Message}> = ({message}) => {
  return (
    <View style={styles.message}>
      <Text style={styles.messageText}>{message.content}</Text>
    </View>
  );
};

export default App;

Can you check if I'm doing something wrong here?

[cdiddy77]

I was able to run it using a different model. Seems to be an issue with model compatibility with the phone used. But the crash shouldn't occur. Error out maybe.

@nadeem-portico which model did you use?

gemma-2b-it-cpu-int8.bin

Hey @nadeem-portico, I'm still getting the error [Error: internal: Failed to initialize engine: %sINTERNAL: ; RET_CHECK failure (external/odml/odml/infra/genai/inference/utils/llm_utils/scoped_file_posix.cc:27) (fd)>=(0)open() failed: /storage/emulated/0/Download/gemma-2b-it-cpu-int8.bin]

import React from 'react';
import {
  ActivityIndicator,
  Keyboard,
  KeyboardAvoidingView,
  Platform,
  Pressable,
  SafeAreaView,
  ScrollView,
  Text,
  TextInput,
  TouchableWithoutFeedback,
  View,
} from 'react-native';
import {colors, styles} from './src/styles';
import {type Message} from './src/types';
import {useLlmInference} from 'react-native-llm-mediapipe';

const samplePrompts = [
  "Explain the difference between 'affect' and 'effect' and use both words correctly in a complex sentence.",
  'If all Roses are flowers and some flowers fade quickly, can it be concluded that some roses fade quickly? Explain your answer.',
  'A shop sells apples for $2 each and bananas for $1 each. If I buy 3 apples and 2 bananas, how much change will I get from a $10 bill?',
  "Describe the process of photosynthesis and explain why it's crucial for life on Earth.",
  'Who was the president of the United States during World War I, and what were the major contributions of his administration during that period?',
  'Discuss the significance of Diwali in Indian culture and how it is celebrated across different regions of India.',
  'Should self-driving cars be programmed to prioritize the lives of pedestrians over the occupants of the car in the event of an unavoidable accident? Discuss the ethical considerations.',
  'Imagine a world where water is more valuable than gold. Describe a day in the life of a trader dealing in water.',
  'Given that you learned about a new scientific discovery that overturns the previously understood mechanism of muscle growth, explain how this might impact current fitness training regimens.',
  'What are the potential benefits and risks of using AI in recruiting and hiring processes, and how can companies mitigate the risks?',
];

let samplePromptIndex = 0;

function App(): React.JSX.Element {
  const textInputRef = React.useRef<TextInput>(null);
  const [prompt, setPrompt] = React.useState('');
  const messagesScrollViewRef = React.useRef<ScrollView>(null);
  const [messages, setMessages] = React.useState<Message[]>([]);
  const [partialResponse, setPartialResponse] = React.useState<Message>();

  const llmInference = useLlmInference({
    storageType: 'file',
    modelPath: '/storage/emulated/0/Download/gemma-2b-it-cpu-int8.bin',
    // 'gemma-1.1-2b-it-gpu-int8.bin' or the name of the model that
    // you placed at android/app/src/main/assets/{MODEL_FILE}
  });

  const onSendPrompt = React.useCallback(async () => {
    if (prompt.length === 0) {
      return;
    }
    setMessages(prev => [...prev, {role: 'user', content: prompt}]);
    setPartialResponse({role: 'assistant', content: ''});
    setPrompt('');
    const response = await llmInference.generateResponse(
      prompt,
      partial => {
        setPartialResponse(prev => ({
          role: 'assistant',
          content: (prev?.content ?? '') + partial,
        }));
      },
      error => {
        console.error(error);
        setMessages(prev => [...prev, {role: 'error', content: `${error}`}]);
        setPartialResponse(undefined);
      },
    );
    setPartialResponse(undefined);
    setMessages(prev => [...prev, {role: 'assistant', content: response}]);
  }, [llmInference, prompt]);

  const onSamplePrompt = React.useCallback(() => {
    setPrompt(samplePrompts[samplePromptIndex++ % samplePrompts.length] ?? '');
    textInputRef.current?.focus();
  }, []);

  return (
    <SafeAreaView style={styles.root}>
      <KeyboardAvoidingView
        keyboardVerticalOffset={0}
        behavior={Platform.OS === 'ios' ? 'padding' : 'height'}
        style={styles.keyboardRoot}>
        <TouchableWithoutFeedback
          onPress={() => Keyboard.dismiss()}
          style={styles.promptInnerContainer}>
          <ScrollView
            ref={messagesScrollViewRef}
            style={styles.messagesScrollView}
            contentContainerStyle={styles.messagesContainer}
            // onContentSizeChange={() =>
            //   messagesScrollViewRef.current?.scrollToEnd()
            // }
          >
            {messages.map((m, index) => (
              <MessageView message={m} key={index} />
            ))}
            {partialResponse && <MessageView message={partialResponse} />}
          </ScrollView>
        </TouchableWithoutFeedback>
        <View style={styles.promptRow}>
          <Pressable
            onPress={onSamplePrompt}
            // disabled={prompt.length === 0 || partialResponse !== undefined}
            style={styles.samplePromptButton}>
            <Text style={styles.samplePromptButtonText}>⚡️</Text>
          </Pressable>
          <TextInput
            ref={textInputRef}
            selectTextOnFocus={true}
            onChangeText={setPrompt}
            value={prompt}
            placeholder={'prompt...'}
            placeholderTextColor={colors.light}
            multiline={true}
            style={styles.promptInput}
          />
          <Pressable
            onPress={onSendPrompt}
            // disabled={prompt.length === 0 || partialResponse !== undefined}
            style={styles.sendButton}>
            {partialResponse !== undefined ? (
              <ActivityIndicator />
            ) : (
              <Text style={styles.sendButtonText}>Send</Text>
            )}
          </Pressable>
        </View>
      </KeyboardAvoidingView>
    </SafeAreaView>
  );
}

const MessageView: React.FC<{message: Message}> = ({message}) => {
  return (
    <View style={styles.message}>
      <Text style={styles.messageText}>{message.content}</Text>
    </View>
  );
};

export default App;

Can you check if I'm doing something wrong here?

I can't help but notice that you are using the int8 model. It's a pretty big file. I have only tested with int4

[alam65]

I'm able to run the app now, my app is not crashing but I'm not getting the response from the model it's just loading. I'm trying to run the example that @cdiddy77 has given in the repo. Any help is appreciated @cdiddy77 @nadeem-portico