Isevisi

Indlela yokuhlanza idatha ye-Spark Streaming
(I) I-DStream ne-RDD
Njengoba sazi, ukubala kwe-Spark Streaming kusekelwe ku-Spark Core, futhi umnyombo we-Spark Core uyi-RDD, ngakho Ukusakazwa kwe-Spark kufanele kuhlobane ne-RDD futhi.Kodwa-ke, ukusakazwa kwe-Spark akuvumeli abasebenzisi ukuthi basebenzise i-RDD ngokuqondile, kodwa kufingqa isethi yemiqondo ye-DStream, i-DStream ne-RDD ubuhlobo obuhlanganisayo, ungakuqonda njengephethini yokuhlobisa ku-Java, okungukuthi, i-DStream iyisithuthukisi se-RDD, kodwa ukuziphatha kufana ne-RDD.
I-DStream ne-RDD zombili zinezimo ezimbalwa.
(1) ibe nezenzo zokuguqula ezifanayo, njengemephu, i-reduceByKey, njll., kodwa futhi nezinye ezihlukile, njenge-Window, i-mapWithSated, njll.
(2) bonke banezenzo Zesenzo, njenge-foreachRDD, ukubala, njll.
Imodeli yokuhlela iyahambisana.
(B) Ukwethulwa kwe-DStream ku-Spark Streaming
I-DStream iqukethe amakilasi ambalwa.
(1) Amakilasi omthombo wedatha, njenge-InputDStream, ethi DirectKafkaInputStream, njll.
(2) Amakilasi okuguqulwa, ngokuvamile i-MappedDStream, ShuffledDStream
(3) amakilasi okukhiphayo, ngokuvamile njenge-ForEachDStream
Kusukela kulokhu okungenhla, idatha kusukela ekuqaleni (okufakiwe) kuye ekugcineni (okuphumayo) kwenziwa ngohlelo lwe-DStream, okusho ukuthi umsebenzisi ngokuvamile akakwazi ukukhiqiza ngokuqondile futhi asebenzise ama-RDD, okusho ukuthi i-DStream inethuba nesibopho sokuba obhekele umjikelezo wempilo wama-RDD.
Ngamanye amazwi, i-Spark Streaming ine-ukuhlanza okuzenzakalelayoumsebenzi.
(iii) Inqubo yokukhiqiza i-RDD ekusakazeni kwe-Spark
Ukugeleza kwempilo kwama-RDD ku-Spark Streaming kunzima kanje.
(1) Ku-InputDStream, idatha etholiwe iguqulelwa ku-RDD, njenge-DirectKafkaInputStream, ekhiqiza i-KafkaRDD.
(2) bese nge-MappedDStream nokunye ukuguqulwa kwedatha, lesi sikhathi sibizwa ngokuqondile ngokuthi i-RDD ehambisana nendlela yemephu yokuguqulwa.
(3) Ekusebenzeni kwekilasi lokuphumayo, kuphela lapho i-RDD iveziwe, ungavumela umsebenzisi ukuthi enze isitoreji esihambisanayo, ezinye izibalo, neminye imisebenzi.